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FOREWORD 


The  Army  is  aware  of  the  increasing  importance  of  training  and  manpower 
estimation  throughout  the  life  cycle  process.  Dollar  and  manpower  require¬ 
ments  are  fixed  very  early  in  system  development  but  their  full  impact  is 
often  not  apparent  until  years  later.  For  this  reason,  improved  cost  and 
training  effectiveness  analysis  (CTEA)  has  become  a  major  Army  need. 

Guidance  exists  for  what  analyses  are  desired,  but  the  state-of-the-art  in 
supporting  technology  has  not  kept  up  with  demands  for  increasingly  precise 
estimates.  As  a  result,  ARI  initiated  a  research  program  on  CTEA  in  1976 
which  was  concerned  with  the  early  estimation  of  training  media,  methods, 
and  costs.  In  1978  the  program  was  expanded  to  include  training  estimation 
problems  during  a  weapon  life  cycle.  Because  the  developmental  history  of 
no  two  systems  is  identical,  the  present  contract  was  designed  too  pull 
together  methods  developed  to  deal  with  individual  cases  and  produce  a 
systematic  procedural  guide  for  an  Army  analyst  tasked  with  CTEA  performance. 
The  result  of  the  effort  is  in  two  volumes.  The  first  examines  CTEA 
requirements  in  the  life  cycle,  the  available  techniques  for  each  analysis 
point,  and  the  R&D  deficiencies.  The  sdcond  is  a  user's  guide  implementing 
the  first  volume  into  a  step-by-step  procedure  for  CTEA  performance  using 
actual  field  data  sources  and  examples.  This  research  is  in  response  to 
Array  project  2Q263742A794  and  special  needs  of  the  Directorate  of  Training 
Developments,  Ft  Bliss,  Texas. 


BRIEF 


Requirement: 

During  the  acquisition  of  Army  materiel  systems  as  defined  by  the  Life  Cycle 
Systems  Management  Model  (LCSMM)  there  Is  a  continuing  requirement  for  Information 
on  the1  cost  and  effectiveness  of  training  personnel  to  operate  and  maintain  the 
system.  At  a  number  of  key  points,  decision  makers  need  reports,  the  result  of 
formal  cost  and  training  analyses  (CTEA).  These  CTEA  evaluate  both  the  development 
of  training  systems  to  support  the  materiel  system  and  the  hardware  to  assess  the 
impact  of  the  system  on  personnel  capabilities.  Army  analysts  currently 
charged  with  conduct  of  CTEA  require  workable  methods  suitable  to  the  particular 
situation  to  perform  these  analyses  In  a  timely  manner  and  to  obtain  reasonable, 
accurate  recommendations  consistently.  Therefore,  this  research  was  undertaken 
with  the  objective  of  providing  Army  analysts  with  a  performance  guide  for  the  con¬ 
duct  of  CTEA  at  each  stage  of  the  LCSMM. 

Procedure: 

Available  methods  were  identified  for  performance  of  CTEA  components.  The  methods 
were  catagorized  using  a  general  CTEA  model  derived  from  key  analyst  questions. 

If  the  answers  to  the  questions  are  negative,  the  analyst  is  routed  to  a  rank 
ordered  set  of  methods  to  generate  the  missing  information.  If  the  answers  are 
positive  the  analyst  is  routed  to  the  next  operation  require^  by  CTEA  regulations. 
The  process  is  continued  until  a  complete  analysis  is  perform..*.  New  methods 
were  generated  where  existing  techniques  proved  inadequate  and  the  complete  process 
incorporated  ir  a  user  guide. 

Findings: 

Systems  may  be  fielded  with  most,  some,  little,  or  none  of  the  data  described  by 
Army  doctrine.  Six  basic  data  conditions  and  sequences  were  found  as  follows: 

(1)  no  task  list  and  no  training  program,  (2)  task  list  but  no  training  program, 
(3)  training  program  but  no  alternatives  and  no  effectiveness  data,  (4)  training 
program  with  effectiveness  data  but  no  alternatives,  (5)  alternative  training 
programs  but  no  effectiveness  data  for  all  alternatives,  and  (6)  training  program 
alternatives  and  effectiveness  data.  Methods  in  current  use  were  found  that  coulc 
contribute  useful  information  covering  one  or  more  of  these  situations  and  were 
within  the  state-of-the-art  of  CTEA  conducted  in  the  Army 

Three  methodological  deficiencies  were  found.  Use  of  historical  data  is 
hampered  because  there  is  no  consolidated  data  bank  or  base  that  describes  tasks 
functionally,  behavioral ly,  and  in  terms  of  training.  Second,  existing  method¬ 
ology  does  not  thoroughly  address  trainability  (e.g..  Can  the  available  personnel 
be  trained  to  perform  the  tasks  at  the  required  level  of  proficiency,  given  the 
proposed  hardware  configuration?).  The  third  gap  was  in  cost  models.  Costs  of 
unit  training  were  omitted  in  even  the  most  comprehensive  methods  examined. 

Utilization  of  Findings: 

A  CTEA  Performance  Guide,  a  fully  proceduralized  manual,  was  prepared  to  provide 
guidance  for  CTEA  analysts.  The  Performance  Guide  contains  step-by-step  instruc¬ 
tions  for  the  selection,  application,  and  implementation  of  the  process  method(s). 
It  includes  guidance  for  synthesizing  methods  on  the  basis  of  inputs  available, 
issues,  and  other  constraints  as  well  as  strategies  for  dealing  with  the  various 
situations.  Instructions  for  using  the  processes  are  complete  making  it 
unnecessary  to  procure  source  documents.  The  procedures  are  illustrated  with 
comprehensive  examples  of  their  use.  The  guide  is  currently  being  tested  in  the 
performance  of  two  Army  CTEAs. 
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COST  AND  TRAINING  EFFECTIVENESS  ANALYSIS  IN  THE  ARMY 
LIFE  CYCLE  SYSTEMS  MANAGEMENT  MODEL 


The  Army's  Life  Cycle  System  Management  Model  (LCSttl)  defines  the  process 
by  which  Army  materiel  systems  are  acquired.  It  was  designed  to  ensure  that  all 
aspects  of  the  system  are  considered  during  the  acquisition  process  including 
personnel  and  training  requirements.  Cost  and  training  effectiveness  analysis 
(CTEA)  provides  to  decision  makers  information  about  how  training  is  influenced 
by  and  influences  the  characteristics  of  the  developing  system.  CTEA  evaluates 
the  development  of  training  systems  to  support  the  materiel  system  and  the  evo¬ 
lution  of  a  materiel  system  compatible  with  personnel  and  training  capabilities. 

OBJECTIVE 

The  objective  of  this  research  was  to  provide  Army  analysts  with  a  perfor¬ 
mance  guide  for  CTEA  at  each  stage  of  the  LCSM.  The  attainment  of  this  objec¬ 
tive  required: 

o  Determination  of  points  in  the  LCSW  at  which  CTEA  are  needed. 

o  Assessment  of  the  utility  of  existing  methods  for  CTEA  at  each 
point. 

o  Adaptation  of  methods  to  each  LCSttl  stage  where  CTEA  are  needed. 

o  Development  of  methods  for  areas  where  existing  ones  are  not 
adequate. 

o  Identification  of  input  information  required  and  available  at 
each  point. 

o  Estimation  of  the  impacts  of  missing  or  degraded  information  on 
the  CTEA. 

o  Synthesis  of  a  general  model  for  CTEA  in  the  LCSKH. 

o  Production  of  a  CTEA  Performance  Guide  for  Army  analysts. 


CTEA  SITUATIONS  AND  STRATEGIES  IN  THE  LCSPW 


Several  LCSft!  events  require  CTEA  information.  For  example,  four  cost  and 
operational  effectiveness  analyses  (COEA)  are  conducted  during  the  LCShW.  CTEA 
support  COEA  by  providing  assessments  of  alternative  ways  to  train  bo  achieve 
the  desired  operational  effectiveness  of  the  system  as  well  as  providing  the 
cost  of  each  proposed  alternative.  At  least  four  CTEA  are  needed  to  support 
the  COEA  and  additional  'CTEA  are  beneficial  to  support  training  development 
decisions. 

Our  examination  of  the  LCSMW  decision  requirements  revealed  the  need  for 
a  more  detailed  analysis  cf  the  CTEA  job  to  be  performed  by  the  Army  analyst. 


Material  systems  do  not  progress  through  the  LCSM1  in  look-step  fashion.  Sys¬ 
tems  may  be  fielded  with  most,  some,  little,  or  none  of  the  data  described  in 
Army  doctrine,  and  the  data  are  not  necessarily  generated  in  the  prescribed 
sequence.  If  we  wished  to  give  the  Army  analyst  the  wherewithal  to  perform  a 
CTEA,  we  had  to  use  as  guidance  (1)  the  decision  to  be  rendered  on  the  basis  of 
the  CTEA  information  and  (2)  the  data  available  to  conduct  the  CTEA.  These  two 
factors  are  the  boundaries  for  CTEA  methodology  regardless  of  the  point  in  the 
LCSf-W. 

Some  data  conditions  and  sequences  are  more  likely  to  occur  than  others. 

For  example,  the  lack  of  a  task  list  or  training  program  implies  the  lack  of 
training  effectiveness  data.  Litton  identified  six  input-data  situations  as: 

1 .  No  task  list  and  no  training  program, 

2.  Task  list  but  no  training  program, 

3.  Training  program  but  no  alternatives  und  no  effectiveness  data, 

4.  Training  program  with  effectiveness  data  but  no  alternatives, 

5.  Alternative  training  programs  but  no  effectiveness  data  for  all 
alternatives,  and 

6.  Training  program  alternatives  and  effectiveness  data  for  all 
alternatives. 

Depending  upon  the  data  situation  the  analyst  needs  a  different  strategy 
to  perform  each  CTEA.  None  of  thj  CTEA  methods  unearthed  in  Lit ton's  literature 
review  provided  all  information  required  for  decisions  at  any  point  in  the  LCSW. 
However,  detailed  examination  of  the  techniques,  steps,  or  processes  within  the 
methods  (i.e.,  the  elements  from  which  the  methods  are  built)  showed  that  each 
method  contributed  to  the  CTEA  job  to  be  performed.  Some  methods  contributed 
more  elements  within  some  data  situations  than  others.  For  example,  informal, 
expert- judgment  methods  in  current  use  generate  task  lists  if  none  are  avail¬ 
able.  Formal  analytic  models  for  prediction  of  training  programs,  however, 
are  superior  tc  the  judgmental  models  if  task  lists  already  exist. 

Litton  devised  a  general  model  for  CTEA  to  guide  the  analyst  in  assessing 
the  data  situation  and  in  selecting  CTEA  processes.  The  model  leads  the  analyst 
through  questions  concerning  the  availability  of  task  lists  and  training  programs, 
including  alternative  training  programs.  Following  these  are  the  estimation  of 
effectiveness,  cost  analysis  of  training  program  alternatives,  cost  effectiveness 
comparisons  of  training  program  alternatives,  and  resolution  of  issues.  Each 
situation  identifies  a  set  of  required  CTEA  processes  and  suggests  an  approach 
to  meet  the  CTEA  objectives.  Litton  designed  strategies  to  guide  the  selection 
of  the  processes  and  conduct  of  the  CTEA. 

SELECTION  OF  CTEA  PROCESS  METHODS 

Powerful  methods  applicable  to  CTEA  and  the  whole  manpower,  personnel,  and 
training  side  of  acquisition  via  the  LCSW  have  been  developed  and  await  bread 
application  and  refinement. 


An  examination  of  the  set  of  CTEA  methods ,  the  constraints  on  them,  and 
the  CTEA  strategies  led  to  the  following  criteria  for  the  selection  of  the 
methods  to  be  included  in  the  CTEA  Performance  Guide: 

o  The  method  must  have  either  demonstrated  or  face  validity. 

o  The  method  selected  must  be  usable  in  the  present  rather  than 
at  some  unspecified  time  in  the  future. 

o  Because  all  six  strategies  are  required  to  meet  the  require¬ 
ments  identified  by  the  general  CTEA  model,  methods  must  be 
selected  or  developed  to  fully  implement  each  strategy. 

In  addition,  alternative  methods  were  selected  for  each  process  required 
by  a  strategy.  Automated  methods  were  selected  whenever  possible  to  meet  con¬ 
straints  that  arise  from  personnel  resources  available  to  perform  CTEA. 

The  methods  that  have  processes  useful  in  the  LCSMM  are: 

o  Training  Efficiency  Estimation  Model  (TEEM;  Jorgensen  and  Hoffer, 

1978), 

o  Training  Consonance  Anaysis  (TCA;  Hawley  and  Thomason,  1978), 

o  Training  Effectiveness,  Cost  Effectiveness  Prediction  (TECEP,  also 
known  as  TAEG  No.  1 6 ;  Braby  et  al.,  1975), 

o  Army  CTEA  Methods  in  Current  Use: 

DIVAD  Gun  CTEA, 

Improved  Hawk  (Hawk  PIP)  Training  Development, 

Roland  Training  Development, 

Improved  TOW  Vehicle  (ITV)  CTEA, 

Diagnostic  Rifle  Marksmanship  Simulators  (DRIMS)  CTEA,  and 

Methods  for  the  Analysis  of  Training  Devices  and  Simulators 

(TRAINVICE) . 

Extant  CTEA  methodology  contained  three  weaknesses.  First,  use  of  histori¬ 
cal  data  was  hampered  by  inadequate  definition  of  analogous  tasks  (i.e.,  tasks  in 
a  fielded  system  functionally  and  behaviorally  similar  to  tasks  in  the  proposed 
system) .  Second ,  the  existing  methodology  did  not  thoroughly  address  the  issue 
of  crainability,  one  of  the  estimations  needed  in  the  LCSMM.  Third,  the  most 
thorough  cost  analysis  model  omitted  costs  of  training  in  units. 

Litton  devised  methods  to  meet  these  methodological  needs.  First,  our 
analogous  task  method  (ATM)  was  developed  to  take  advantage  of  information  ob¬ 
tained  in  the  process  of  training  soldiers  on  fielded  weapons  systems.  The 
metnoa  is  applied  on  a  task-by-ta3k  basis  with  an  overall  estimate  of  training 
effectiveness.  The  analogous  task  method  has  six  steps:  (1)  definition  of  the 


critical  tasks  to  be  performed  on  the  developing  system  (the  target  tasks);  (2) 
classification  of  the  target  tasks  to  provide  a  basis  for  finding  the  analogous 
tasks;  (3)  analogous  task  identification  and  selection;  (4)  assessment  of  train¬ 
ing  for  the  analogous  tasks;  (5)  generation  of  estmates  of  training  for  the  tar¬ 
get  tasks;  and  (6)  aggregation  of  the  effectiveness  and  cost  measures  across  all 
tasks  to  obtain  a  picture  of  training  for  the  developing  system  as  a  whole.  ATM 
is  illustrated  in  Figure  II-2.  Each  rectangle  represents  a  step  in  the  process, 
while  the  processes  indicated  in  the  circles  show  the  relationship  of  the  process 
to  other  processes  in  a  general  CTEA  model  to  be  presented  later. 

The  second  methodology  devised  by  Litton,  trainability  analysis,  is  needed 
to  examine  the  interactions  among  tasks,  training  program  alternatives,  and 
personnel  characteristics.  Once  tasks  have  been  identified  and  alternative 
means  of  training  those  tasks  have  been  predicted  or  developed,  it  is  necessary 
to  determine  that,  given  the  characteristics  of  the  personnel  who  will  man  the 
system,  the  tasks  can  be  trained  to  required  levels  of  proficiency.  Our  litera¬ 
ture  review  unearthed  no  explicit  method  for  trainability  analysis,  therefore 
we  proposed  a  process  to  use  as  an  interim  technique  until  further  research  is 
conducted. 

Litton* s  third  methodological  development  expanded  the  TECEP  cost  model  to 
cover  costs  of  institutional  field  training  and  training  in  Army  units.  The 
Litton  cost  model  is  intended  to  aid  the  CTEA  analyst  in  preparing  recoranenda- 
tions  to  the  decision  maker  regarding  choices  among  alternatives.  It  is  not 
intended  for  budgetary  purposes.  The  Litton  model  does,  however,  capture  perti¬ 
nent  institution  and  unit  costs. 

Four  methods  developed  for  the  Air  Force  were  considered  but  not  incorporated 
in  the  model  at  present.  They  require  substantial  revision  to  make  them  suitable 
for  Army  use.  These  methods  are  automated  in  total  or  in  part  and  use  consolidated 
data  bases.  They  are: 

o  Coordinated  Human  Resources  Technology  (CHRT;  Goclowski  et  al., 

1978), 

c  Method  of  Designing  Instructional  Alternative  (MODIA:  Carpenter- 
Hufftoan  et  al.,  1977), 

o  Digital  Avionics  Information  System:  Training  Requirements  Analysis 
Model  (DAIS/TRAMOD) ;  Czuchry  et  al.,  1978), 

o  B-1  Bomber  Systems  Approach  to  Training  (B-1  SAT;  Sugarman  et  ad., 

1975). 

The  Training  Developer’s  Decision  Aid  (Pieper  et  al.,  1979)  method  was  de¬ 
veloped  for  the  Array  and  it  is  generally  applicable  to  CTEA.  It  has  not  yet 
been  fully  developed,  however,  and  seme  problems  in  implementation  remain, 
especially  the  generation  of  task  lists.  Because  of  the  extent  of  arbitration 
in  the  logic  of  the  method,  an  extensive  validation  effort  is  necessary  to 
demonstrate  the  usefulness  and  precision  of  its  outputs.  Available  descrip¬ 
tions  of  the  method  are  not  complete  enough  to  permit  its  inclusion  in  the 
CTEA  Performance  Guide. 
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SPECIFICATION  FOR  THE  CTEA  PERFORMANCE  GUIDE 


The  means  for  synthesizing  CTEA  methods  from  available  process  methods 
have  been  provided,  but  guidance  for  the  application  of  those  means  as  well  as 
for  the  implementation  of  the  selected  process  methods  must  also  be  provided. 
The  CTEA  Performance  Guide  (a  separate  voltme)  provides  such  guidance. 

The  CTEA  Performance  Guide:  , 

o  Contains  guidance  for  conducting  CTEA  on  the  basis  of  inputs  avail¬ 
able  for  analysis,  issues  to  be  resolved,  and  other  constraints  on 
the  analysis,  such  as  the  skills  and  numbers  of  CTEA  personnel  and 
the  time  available  for  performance. 

o  Contains  strategies  for  dealing  with  the  various  input-data  situa¬ 
tions  that  are  a  function  of  both  the  phasing  of  LCSMi  itself  and 
the  deviations  from  the  LCSf-W  that  any  particular  acquisition  pro¬ 
gram  exhibits . 

o  Contains  complete  instructions  for  each  process  method  so  that  the 
unavailability  of  source  documents  will  not  impede  the  conduct  of 
a  CTEA. 

o  Provides  examples  or  illustrations  of  the  applications  of  process 
methods . 


GENERAL  RECOMMENDATIONS 


Results  of  the  CTEA  literature  review,  synthesis  of  methods  and  guidebook 
preparation  lead  to  the  following  recommendations: 

o  Program  TEEM,  TCA,  and  the  Litton  Cost  Model  before  the  CTEA 
Performance  Guide  is  released  to  CTEA  analysts.  Be  sure  that 
programs,  suitable  machines,  and  reasonable  interfaces  with  CTEA 
analysts  are  provided. 

o  Develop  and  adapt  one  or  more  of  the  automated  Air  Force  methods 
discussed  above  for  Array  Training  Developments  and  CTEA. 

o  Develop  means,  including  explicit  contractual  requirements,  to 
assure  that  all  developmental  data  needed  by  CTEA  analysts  become 
available  in  timely  fashion. 

o  In  CTEA  study  directives,  include  specifications  of  CTEA  timing 
or  phasing  in  conformity  with  the  CTEA  locations  described  in 
Section  II  of  this  report. 

o  In  CTEA  study  directives,  specify  CTEA  requirements  on  the  basis 
of  a  realistic  appraisal  of  the  actual  status  of  the  acquisition 
program  of  interest  rather  than  by  reference  to  the  LCSMM. 
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SECTION  I 
INTRODUCTION 


The  acquisition  process  for  military  weapon  systems  moves  through  several 
phases  beginning  with  the  formulation  and  examination  of  alternative  concepts 
for  achieving  desired  ends  and  finishing  with  deployment  of  a  system  with  demon¬ 
strated  capability  to  achieve  those  ends  at  an  acceptable  cost.  That  is,  the 
acquisition  process  ends  with  such  a  system  if  at  every  step  the  right  decisions 
are  made.  Making  the  right  decisions  depends  on  timely  and  valid  information 
including  information  concerning  cost  and  effectiveness  of  training  programs  to 
support  weapon  system  operation  and  maintenance.  Cost  and  training  effectiveness 
analysis  (CTEA)  provides  that  training  system  information. 

CTEA  have  long  been  performed  for  existing  weapon  systems  toward  the  same 
purpose:  organization  and  presentation  of  information  to  enable  decision  makers 
to  select  from  among  alternative  training  approaches  those  that  increase  man- 
machine  system  performance  within  cost  constraints.  Weapon  system  training  ef¬ 
fectiveness  analysis  identifies  training  problems  and  proposes  solutions  to 
those  problems. 

The  Army's  Life  Cycle  System  Management  Model  (LCSM1),  the  process  by  which 
the  Army  materiel  systems  are  acquired,  was  designed  to  ensure  that  ell  aspects 
of  the  weapon  system  are  considered  during  the  acquisition  process.  These  as¬ 
pects  include  integration  of  user  and  trainer  requirements  and  developments  with 
the  parent  weapon  system  development.  Moreover,  it  was  intended  to  allow  the 
trainer  to  consider  cost-effective  training  options  before  final  weapon  system 
fielding.  However,  the  emphasis  in  the  LCSNW  has  been  on  events  concerned  with 
the  parent  system. 

It  is  essential  that  the  personnel  and  training  impacts  be  known  earl;’. 
Therefore,  questions  of  estimated  training  program  effectiveness  and  estimated 
cost  need  early  answers.  The  answers  are  highly  sensitive  to  changes  in  the 
development  of  the  parent  system.  As  with  the  parent  system,  testing  is  needed 
for  trade-offs  between  training  on  operational  equipment,  training  simulators 
and  devices,  and  other  training  issues.  Frequently  no  such  testing  of  simula¬ 
tion,  training  equipment,  or  training  programs  is  accomplished.  When  testing 
is  done,  it  occurs  late  in  the  cycle  or  after  the  system  is  fielded.  Late 
testing  fails  to  ensure  cost-effective  training:  the  user  should  be  confirming 
the  effectiveness  of  training  during  late  stages,  not  starting  its  assessment. 

The  training  developer  lacks  guidelines  and  procedures  for  the  concurrent 
development  of  training  programs  with  the  development  of  the  parent  system.  The 
problem,  then,  is  to. furnish  a  complete,  detailed  roadmap  on  the  best  way  to  in¬ 
tegrate  training  requirements  into  the  materiel  development  cycle  and  provide 
tools  for  trade-off  decisions  at  each  critical  training  program  decision  point. 
The  purpose  of  this  research  is  to  provide  the  CTEA  analyst  with  tools  to  organ¬ 
ize,  select,  and  analyze  data  for  decision  making. 

CTEA  methodology  in  the  LCSNM  must  adapt  to  an  array  of  data  types  that  vary 
in  precision,  accuracy,  ana  completeness.  Some  CTEA  methods  require  empirical 
performance  data,  others  human  factors  information,  and  still  others  require 
only  task  descriptions.  These  three  CTEA  styles  represent  a  continuum  in  data 


requirements.  CTEA  methodology  applicable  in  the  LCSM1  cannot  be  drawn  exclu¬ 
sively  from  any  one  of  these  styles.  It  must  synthesize  methods  suited  to  the 
types  of  data  available  throughout  the  LCSM  and  responsive  to  information  needed 
for  decision  making.  CTEA  in  the  LCSM  needs  methodology  capable  of  replacing 
input  data  at  successive  stages  as  more  definite  data  become  available.  The 
CTEA  methodology  must  be  iterative  and  usable  as  a  tool  in  cost-effectiveness 
estimation  and  in  hypothesis  testing.  At  each  application,  training  require¬ 
ments  oust  be  forecast,  alternatives  that  offer  probable  cost-effectiveness 
selected,  and  results  of  prior  estimations  tested  to  gradually  refine  and  im¬ 
prove  training. 

CTEA  is  a  method  for  defining  training  alternatives,  comparing  those  al¬ 
ternatives,  and  defining  the  proficiency  obtainable  with  each  training-hardware 
combination.  That  is,  CTEA  must  be: 

o  prescriptive  -  identify  training  program  elements  likely  to  be 
effective; 

o  diagnostic  -  identify  weaknesses  in  training  programs;  and 

o  predictive  -  define  the  probable  effectiveness  and  cost  of  training 
programs. 

CTEA  methodology,  therefore,  is  more  than  a  means  for  differentiating  among 
alternatives  (the  hypothesis  testing  function) .  At  early  stages  of  the  acquisi¬ 
tion  cycle,  CTEA  methods  need  to  identify  existing  data  to  define  training  op¬ 
tions.  In  this  forecasting  role,  CTEA  predicts  an  array  of  training  alternatives. 
CTEA  must  reveal  training  deficiencies  and  reconmend  solutions.  In  late  stages, 
CTEA  methods  must  continue  forecasting  and  diagnosis  and  must  differentiate  among 
alternatives.  Changes  in  threat,  tactics,  and  operations  continually  influence 
system  performance.  CTEA  methodology  must  be  capable  of  continually  assessing 
these  impacts  on  effectiveness,  identifying  corrective  options,  and  guiding 
modifications. 

In  summary,  the  objectives  of  CTEA  in  the  LCSNW  are  to: 

o  Ensure  that  training  processes  are  initiated  early  in  the  LCSMM 
and  are  accomplished  in  parallel  with  combat  developments 
processes; 

o  Ensure  that  a  training  subsystem  is  developed  with  the  same 
degree  of  scientific  application  as  is  the  parent  hardware 
system; 

o  Ensure  that  training  system  alternatives  are  considered;  and 

o  Provide  decision  makers  with  information  at  critical  points 
concerning  the  training  and  hardware  system. 

Research  Objective  and  Overview 

The  objective  of  this  research  was  to  provide  Army  analysts  with  a  perfor¬ 
mance  guide  for  CTEA  at  each  stage  of  the  LCSMM.  Figure  1-1  represents  an  over¬ 
view  of  the  conduct  of  this  research.  A  description  of  the  acquisition  process, 


Figure  1-1 


OVERVIEW  OF  CTEA  RESEARCH 


the  Army's  LCSM4,  was  first  developed.  From  the  description  of  the  LCSMM  were 
derived  locations  of  CTEA.  The  CTEA  locations  imply  the  probable  input  and  out¬ 
put  information  of  the  CTEA,  such  as  CTEA  output  required  for  Cost  and  Opera¬ 
tional  Effectiveness  Analysis  (COEA).  Existing  CTEA  methods,  or  methods  applic¬ 
able  to  CTEA  objectives,  were  reviewed  to  determine  their  suitability  for  CTEA 
during  weapon  system  development.  In  addition,  CTEA  methods  were  developed  for 
areas  that  existing  methodology  failed  to  cover  adequately.  The  result  is  a 
general  CTEA  model  that  synthesizes  a  family  of  CTEA  methods,  each  of  which  is 
applicable  at  a  different  point  in  the  LCSM.  The  methods  that  appeared  most 
applicable  were  selected  for  inclusion  in  the  user's  manual.  The  selected 
methods  were  written  in  procedural  terms  and  were  demonstrated  using  examples 
gleaned  from  Army  analyses  of  developing  weapon  systems. 

Organization  of  the  Report 

Section  II  of  this  report  contains  an  overview  of  the  LCSM4.  an  examination 
of  the  requirements  and  purposes  of  CTEA  within  the  LCSftl,  and  recommended  loca¬ 
tions  for  CTEA  in  the  LCSM1.  This  section  defines  the  problems  of  CTEA  for 
which  solutions  are  proposed  in  the  following  section. 

Section  III  summarizes  existing  CTEA  and  CTEA-applicable  methods. 

Section  IV  presents  methods  devised  by  Litton  during  the  present  research 
to  augment  existing  methods. 

Section  V  discusses  a  general  CTEA  model  for  use  in  the  LCSW  and  synthe¬ 
sizes  the  material  selected  for  the  user's  manual.  It  examines  the  constraints 
on  employment  of  CTEA  methods,  and  explains  CTEA  strategies. 

Section  VI  offers  conclusions  and  recommendations.  It  introduces  a  com¬ 
panion  docunent,  The  Cost  yd  Training  Effectiveness  Analysis  Performance  Guide, 
which  contains  procedures  for  ClEA  in  the  LCShM  and  examples  of  their  use. 
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SECTION  II 


COST  AND  TRAINING  EFFECTIVENESS  ANALYSIS  IN  THE 
LIFE  CYCLE  SYSTEM  MANAGEMENT  MODEL 


The  process  through  which  Army  materiel  systems  are  acquired  is  described 
by  the  Life  Cycle  System  Management  Model  (LCSm) ,  expressed  in  a  119  event 
flowchart  (Department  of  the  Army,  1975).  LCSMM  outlines  procedures  for  the 
development  and  acquisition  of  Army  systems  from  concept  investigation  through 
ultimate  disposal  of  obsolete  systems.  It  covers  coordination  of  combat  develop¬ 
ment,  research  and  development,  production  and  logistic  support,  training  and 
personnel  requirements,  and  actions  required  to  develop  and  maintain  the  system. 

The  Integrated  Personnel  Support  (IPS)  system  is  the  process  by  which  per¬ 
sonnel  considerations  are  integrated  into  the  LCSMM  (Department  of  the  Army, 

1978).  The  goal  of  IPS  is  oO  ensure  that  personnel  resources  are  planned,  de¬ 
veloped,  acquired,  tested,  and  deployed  in  conjunction  with  the  materiel  acqui¬ 
sition  process.  Personnel  resources  include  the  number  and  characteristics  of 
personnel  required  to  operate,  support,  and  maintain  the  system,  their  training, 
interface  with  the  hardware,  huaan  resources  development,  and  other  personnel 
factors . 

The  military  services  conduct  continuing  analyses  of  the  threat  and  asso¬ 
ciated  friendly  mission  areas,  including  military  mission  needs,  capabilities, 
resources,  and  technology,  .ased  on  their  analyses,  the  services  prepare  mission 
element  need  statements  (MENS)  for  approval  by  the  Secretary  of  Defense  to  jus¬ 
tify  major  new  system  acquisition.  MENS  contain  a  "constraints"  section  for  the 
manpower  and  resources  for  the  proposed  system.  Minimization  of  the  number  and 
skill  requirements  of  people  needed  to  operate  and  support  the  proposed  system 
is  a  goal  of  the  planning.  Thus,  personnel  estimates  such  as  the  number,  skills, 
training,  and  occupations  required  are  part  of  the  MENS. 

The  number  and  characteristics  of  personnel  estimated  for  the  system  influ¬ 
ence  task  and  training  subsystem  design.  The  training  must  accomnodate  the 
number  of  personnel,  their  aptitude  levels,  and  prior  training.  The  type  of 
training  depends  upon  the  nature  of  the  personnel  who  are  to  be  trained. 

Approval  of  the  MENS  by  the  Secretary  of  Defense  authorizes  the  Army  to 
initiate  the  LCSMM.  The  process  consists  of  four  major  phases:  Conceptual, 
Demonstration  and  Validation,  Full-Scale  Development,  and  Production  and  Deploy¬ 
ment.  Specificity  of  the  materiel  design,  personnel  considerations,  and  train¬ 
ing  vary  with  the  LCSNM  phases.  Early  in  the  cycle,  such  as  the  Conceptual 
phase,  personnel  estimates  and  information  on  training  effectiveness  are  gleaned 
from  engineering  data  and  general  principles  of  human  factors  and  instructional 
design.  In  later  phases  empirical  data  are  available  from  developmental  and 
operational  tests.  During  deployment,  cost  and  effectiveness  data  are  collected 
in  field  tests.  The  data  increase  in  accuracy,  objectivity,  scope,  and  complete¬ 
ness  as  the  acquisition  cycle  proceeds. 

Several  L.CSW  events  require  CTEA  information  and  are  influenced  by  CTEA. 
Figure  II- 1  shows  the  LCSMM  locations  that  produce  new  or  revised  input  data  for 
CTEA  and  that  require  CTEA.  For  example,  four  cost  and  operational  effective¬ 
ness  analyses  (COEA)  are  conducted,  one  in  each  major  phase.  CTEA  support  COEA 
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by  providing  assessments  of  alternative  ways  to  train  to  achieve  the  desired 
operational  effectiveness  cf  the  system  (Department  of  the  Army,  1977).  At 
least  four  CTEA  are  needed  to  support  the  COEA,  and,  according  to  the  Army  regu- 
-#  lations  cited  in  this  section,  additional  CTEA  are  beneficial  to  support  train¬ 

ing  development  decisions. 

If  CTEA  locations  in  the  LCSW  are  bound  by  events  that  provide  new  or 
revised  data  to  CTEA  and  by  events  that  require,  or  are  influenced  by  CTEA,  then 
six  CTEA  are  recom&anded,  as  shown  in  Table  II- 1.  Two  CTEA  are  recoamended 
during  each  of  the  first  two  phases,  Conceptual  and  Demonstration  and  Valida¬ 
tion.  One  CTEA  is  recommended  during  each  of  the  two  later  phases,  Full-Scale 
Development  and  Production  and  Deployment.  The  events  that  bound  the  CTEA  and 
reasons  for  the  reconnended  CTEA  locations  are  described  in  the  remainder  of 
this  section. 

A.  Conceptual  Phase 

Approval  of  the  MENS  by  the  Secretary  of  Defense  authorizes  the  Army  to 
initiate  the  Conceptual  Phase.  This  phase  establishes  the  technical,  military, 
and  economic  bases  for  the  proposed  system  and  formulates  concepts  through  re¬ 
search  and  the  development  of  experimental  prototypes.  It  includes  the  solici¬ 
tation,  evaluation,  and  competitive  exploration  of  alternative  means  of  achieving 
the  mission. 

Several  requirements  documents  are  prepared  during  this  phase  that  need 
information  concerning  personnel  and  training.  An  early  training  requirements 
docunent  is  the  Task  and  Skill  Analysis  (TASA)  that  describes  roughly  what  the 
people  need  to  do  to  operate  and  maintain  the  system.  Ideally,  the  TASA  con¬ 
tains  three  categories  of  tasks:  machine  functions,  human  functions,  and  shared 
functions.  Critical  tasks  are  identified  in  the  human  and  shared  function  cate¬ 
gories.  The  critical  tasks  are  likely  to  require  formal  training  and  they  guide 
the  training  support  plan.  At  this  early  stage  the  training  support  plan  focuses 
on  training  factors  that  influence  hardware  and  training  device  designs  to  anti¬ 
cipate  potential  trade-offs.  The  Outline  Individual-Collective  Training  Plan 
(OICTP),  an  element  of  the  training  support  plan,  summarizes  expected  courses 
of  instruction,  changes  in  existing  courses,  and  training  and  performance  aids 
for  the  new  system. 

Personnel  requirements  are  based  on  the  rough  TASA.  They  include  estimates 
of  the  nunber  of  personnel  required,  the  skills  needed,  unique  mental  and  physi¬ 
cal  requirements,  and  human  resources  development  factors  (e.g.,  morale,  organi¬ 
zational  climate) .  Personnel  requirements  and  savings  are  included  as  part  of 
the  prospective  operational  effectiveness  and  costs  in  the  Letter  of  Agreement 
(LOA). 

The  LOA  is  a  docunent  jointly  prepared  and  authenticated  by  the  combat  de¬ 
veloper'  and  the  materiel  developer.  It  outlines  their  basic  agreements  for  ad¬ 
vanced  development  investigations  of  the  proposed  system.  Agreements  and  in¬ 
vestigations  cover  operational,  technical,  personnel,  training,  and  logistic 
support  concepts.  Army  regulations  demand  quantitative  estimates  in  the  LOA  of 
operational  effectiveness  and  cost  of  the  new  system  (Department  of  the  Army, 
1979).  Thus,  cost  and  effectiveness  information  is  required  preceding  the  LOA. 
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Table  IX-1.  CTEA  in  cha  LCSMM 


LCSMM 

PHASE 

CTEA 

PRIMARY  LCSMM  EVENTS 
YIELDING  DATA  TO 

CTEA 

ISSUES  REQUIRING 

RESOLUTION  ! 

CONCEPTUAL 

I 

e  MENS 

e  INITIATION  OF  TRAIN¬ 
ING  PLANNING 

e  INITIATION  OF  LOGISTICS 
SUPPORT  PLANNING 

e  TRAINABILITY  OF  l 

BASIC  CONCEPT  1 

e  COST  OF  TRAINING 
e  TRAINING  PROGRAM 

ELEMENTS  TO  BE  IN-  j 

CLUDED  OR  STUDIED 

I  A 

(update) 

e  LOA 

e  ORGANIZATIONAL  AND 
OPERATIONAL  CONCEPTS 

e  TRAINABILITY  OF  | 

ALTERNATIVE  CONCEPTS 
e  RELATIVE  COST 

EFFECTIVENESS  OF  | 

TRAINING  PROGRAMS  ' 

OF  ALTERNATIVE 

CONCEFTS 

DEMONS  TRATI ON 
AND 

VALIDATION 

II 

e  OAP 
•  DT/OT  I 

i 

e  TRAINABILITY 
e  NECESSARY  REVISIONS 

OF  TRAINING  PROGRAMS  ( 
e  RELATIVE  COST 
EFFECTIVENESS 

II  A 
(update) 

e  PQQPRI 
e  DCP(IPS) 

e  TRAINABILITY 
e  TDR 

e  RELATIVE  COST 
EFFECTIVENESS  OF 

REVISED  TRAINING 
PROGRAMS 

FULL-SCALE 

ENGINEERING 

DEVELOPMENT 

III 

• 

e  NET  PLAN 
e  AP 

e  DT/OT  II 

1 

e  PERFORMANCE  VERSUS 
STANDARDS 

e  PERFORMANCE  VERSUS  j 

HARDWARE  AND  TRAINING  1 
PROGRAM  DESIGN, 
PERSONNEL  SELECTION,  . 

ETC.  1 

PRODUCTION 

AND 

DEPLOYMENT 

IV 

(update) 

e  TRAINING  PLAN  UPDATE 
e  DRAFT  TRAINING 

PROGRAM 
e  AP 

e  DT/OT  III 

e  COST  EFFECTIVESS  OF 
REVISED  TRAINING  j 

PROGRAM 

_ 1 

Information  in  the  LOA  serves  as  a  basis  for  ensuing  documents  including 
Organizational  and  Operational  Concepts,  Outline  Acquisition  Plan,  Qualitative 
and  Quantitative  Personnel  Requirements  Information,  and  Basis  of  Issue  Plan, 

c  The  Organizational  and  Operational  Concepts  document  describes 
trade-offs  between  organizational  and  operational  equipment  and 
personnel . 

o  The  Qualitative  and  Quantitative  Personnel  Requirements  Information 
is  issued  first  in  tentative  or  provisional  form  (TQQPRI  or  PQQPRI). 

It  provides  information  concerning  the  number  and  skills  of  personnel 
required  for  the  operation,  support,  and  maintenance  of  the  proposed 
system. 

o  The  Basis  of  Issue  Plan  (BOIP)  indicates  the  quantity  of  new  or 
modified  equipment  for  each  type  of  organization  and  changes  in 
personnel  and  supporting  equipment. 

The  next  major  document  is  the  Concept  Formulation  Fackage  (CFP)  which 
documents  that  the  objectives  of  the  system  concept  are  met.  The  CFP  describes 
the  technical  approach,  trade-offs,  risks,  personnel  requirements,  costs,  sched¬ 
ules,  and  logistic  support  requirements.  It  includes  the  results  of  a  COEA. 
Thus,  a  CTEA  is  needed  to  provide  information  assessing  alternative  ways  of 
training  to  achieve  the  desired  operational  objective.  Baseline  Cost  Estimates 
(BCE)  that  document  the  cost  of  acquisition  and  operation,  including  personnel 
costs,  are  developed  for  the  CFP  and  are  refined  throughout  the  acquisition 
cycle. 

The  Outline  Acquisition  Plan  (OAP),  formerly  called  the  Development  Plan, 
contains  plans  for  the  management  of  the  advanced  development  of  the  system. 

The  materiel  developer  prepares  the  OAP  using  input  from  the  LOA,  CPF,  ana 
ICTP.  TRADOC,  ADMINCEN,  and  LOGCEN  assist  with  personnel  issues  that  require 
testing  and  to  identify  training  requirements.  Researcn  on  personnel  trade¬ 
offs  and  requirements  is  conducted  and  the  results  are  incorporated  into  the 
OAP. 


The  Conceptual  phase  culminates  with  Milestone  I,  the  first  In-Process 
Review  or  Army  Systems  Acquisition  Review  Council  and  Defense  System  Acquisition 
Review  Council  (ASARC/DSARC  I)  authority  to  proceed  with  the  Demonstration  and 
Validation  phase. 

CTEA  Implications 

CTEA  in  the  Conceptual  phase  need  to  provide  a  deliberate  assessment  and 
analysis  of  alternative  means  of  training  to  achieve  the  operational  effective¬ 
ness  of  materiel  systems  within  a  force.  Tasks  essential  to  mission  accomplish¬ 
ment  with  alternative  system  designs  vary  in  the  difficulty  with  which  they  are 
performed  and  trained.  Overall  operational  effectiveness  of  the  system  varies 
as  a  result.  An  accurate  determination  of  operational  effectiveness  must,  be 
based  on  the  proficiency  likely  to  be  obtained  with  each  design  and  the  cost  of 
training  necessary  to  achieve  that  proficiency.  The  CTEA  should  provide  that 
information  for  eacn  of  the  materiel  system  design  alternatives. 


A  preliminary  cost  and  operational  effectiveness  analysis  is  a  portion  of 
the  LOA.  This  analysis  should  "demonstrate  that  there  is  a  reasonable  chance 
of  providing  a  significant  advance  in  operational  capability  at  an  acceptable 
cost"  and  should  "indicate  probable  improvement  in  force  effectiveness,  probable 
relative  coats  and  the  military  need  for  the  proposed  system"  (Department  of  the 
Army,  1977).  Moreover,  it  should  "outline  the  methods  and  costs  involved  in  ob¬ 
taining  information  on  critical  issues  for  the  CFP  COEA"  (ibid). 

This  preliminary  CTEA  is  important  in  defining  the  system  concept.  It  com¬ 
pares  the  training  effectiveness  and  costs  of  high-risk  training  elements  for 
each  of  the  alternative  materiel  systems  being  explored.  It  identifies  the 
high-risk  and  high-cost  components  as  early  as  possible  in  the  LCSM4.  It  en¬ 
sures  a  complete  OICTP  by  providing  known  training  concepts,  strategies,  costs, 
and  requirements.  This  CTEA  is  prescriptive,  stating: 

o  Unique  performance  requirements  that  limit  system  effectiveness 
because  of  low  probability  of  trainability; 

V' 

o  Training  program  elements  that  must  be  included  in  the  training 
system  to  enable  fielding  of  a  fully  effective  weapon  system*  and 

». 

o  Potential  impact  of  training  program  omissions  because  of  cost 
and  development  risk. 

The  CTEA  should  project  the  consequences  of  various  outcomes  in  a  "train- 
ability  risk  analysis"  and  their  costs.  Where  alternative  systems  are  being 
considered,  the  cost  and  trainability  risk  figure  into  comparison  of  the  alter¬ 
natives.  Trainability  issues  a re: 

o  What  task  groups  are  exceptionally  complex  and  constitute  high 
training  development  risk? 

o  What  are  the  standards  of  performance  for  the  high  risk  tasks? 

o  Which  high  risk  tasks  require  above  average  mental  and  physical 
ability? 

o  What  are  the  difficulties  in  training  the  target  population  for 
each  alternative  technical  approach? 

o  What  training  approaches  offer  the  greatest  potential  for  solving 
each  of  the  high  risk  training  problems? 

o  What  are  the  high  risk  training  support  requirements  for  each 
alternative  system? 

c  What  are  the  costs  of  the  high  risk  training  support  requirements? 

A  variety  of  input  data  for  the  CTFA  is  likely  to  be  available.  The  MENS 
specifies  the  military  need  and  constraints  on  the  manpower  and  resources.  A 
basic  system  concept,  or  alternative  system  concepts,  has  oeen  expressed  and 
the  TASA  and  OICTP  provide  a  rough  estimate  of  the  tasks  and  training  summary, 
including  the  identification  of  high-risk  tasks,  decisions  on  the  locations  fcr 
training,  ana  general  statements  about  the  type  of  training  to  he' conducted. 
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The  output  of  this  preliminary  CTE/  needs  to  define  the  trainability  of 
each  alternative  given  the  outline  training  program,  provide  the  cost  of  train¬ 
ing  for  each  BCE,  and  provide  recommendations  for  training  program  elements  to 
be  included  in  the  OICTP  or  examined  in  research.  It  should  state  the  training 
issues  to  be  examined  and  provide  guidance  for  detailed  front  end  analysis  for 
training  program  refinement. 

An  update  of  the  CTEA  is  recotmended  as  input  to  the  COEA  that  supports 
the  CFP.  The  CFP  is  the  culmination  of  research  designed  to  identify  alterna¬ 
tive  means  of  satisfying  the  needs  set  forth  in  the  MENS.  The  materiel  developer 
performs  engineering  analyses,  identifies  technical  approaches  and  their  support, 
technical  and  logistical  concepts  including  the  provisional  QQPRI.  The  combat 
developer  formulates  the  BOIP,  and  together  the  materiel  and  combat  developers 
formulate  the  Organizational  and  Operational  Concepts.  The  CFP  is  a  compilation 
of  these  documents  that  consider  materiel  and  personnel  issues,  screening  of 
alternative  systems  concepts,  trade-off  analyses,  and  a  COEA. 

The  CTEA  at  the  CFP  stage  addresses  the  relative  effectiveness  of  alterna¬ 
tive  training  programs.  That  is,  given  the  proposed  training  program  for  each 
alternative  materiel  approach,  is  it  likely  that  desired  weapon  system  opera¬ 
tional  capability  can  be  achieved?  The  alternative  with  the  highest  trainability 
estimate  is,  from  the  point  of  view  of  training  effectiveness,  the  best  techni¬ 
cal  alternative. 

The  CFP  identifies  those  system  concepts  deemed  suitable  for  demonstration 
and  validation.  The  concepts  represent  a  single  option,  alternative  systems,  or 
alternative!  subsystems.  The  options  may  include  program  improvement  recommenda¬ 
tions  for  existing  systems,  purchase  of  systems  from  other  services  or  other 
countries,  or  new  development.  The  CTEA  methodology  must  deal  with  all  of  these 
contingencies . 

Training  effectiveness  analysis  at  the  CFP  Stage  must  consider  the  proba¬ 
bility  of  training  success  given  current  knowledge  concerning  the  efficacy  of 
various  training  methods  for  various  types  of  tasks.  The  CTEA  should  appraise 
the  probable  effectiveness  of  the  training  programs  proposed  for  alternative 
system  concepts,  identify  the  strengths  and  weaknesses  of  the  programs,  and 
supply  cost  information  for  comparisons. 

Data  available  include  the  background  information  for  the  CFP:  the  MENS, 
Organizational  and  Operational  Concepts,  CAP,  QQPRI,  EOIP,  LOA,  and  input  on 
operational  costs  and  effectiveness  for  the  COEA.  The  data  may  include  results 
of  research  on  issues,  front-end  analysis  of  prototypes,  and  an  update  of  sys¬ 
tem  functional  requirements.  The  TASA  may  be  updated,  and  high-risk  tasks  may 
be  defined.  Required  output  from  the  CTEA  includes  information  on  the  cost  and 
trainability  of  alternative  systems  to  support  the  selection  of  the  best  tech¬ 
nical  approach  and  recommendations  of  ways  to  enhance  training  effectiveness 
of  selected  system  alternatives. 

In  the  Conceptual  phase,  two  CTEA  have  been  recomnended .  The  first  precedes 
the  LOA  and  provides  initial  estimates  of  costs  and  effectiveness  of  training 
based  on  very  rough  task  statements  and  personnel  estimates.  The  second  uses 
the  information  compiled  for  the  LCA  arid  additional  information  prepared  for  the 
CFP.  These  CTEA  are  required  in  the  ideal  case.  In  reality  personnel  and  train¬ 
ing  considerations  lag  behind  hardware  development.  Thus,  task  and  training  data 
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may  not  be  updated  between  the  LQA  and  CFP  and  as  a  result  there  may  be  no  new 
data  for  the  second  CTEA.  This  discussion,  however,  presents  the  ideal  case 
since  the  goal  is  to  provide  tools  to  perform  the  CTEA  as  they  should  be  done. 

B.  Demonstration  and  Validation  Phase 

A  positive  Milestone  I  decision  allows  the  Army  to  proceed  with  the  Demon¬ 
stration  and  Validation  phase.  This  phase  consists  of  the  steps  needed  to 
resolve  conflicts  identified  during  the  Conceptual  phase,  verify  preliminary 
designs  and  engineering,  accomplish  necessary  planning,  fully  analyze  trade-off 
proposals,  and  prepare  the  contracts  required  for  full-scale  development.  This 
phase  includes  the  use  of  developmental  prototypes  in  development  and  operational 
tests  (DT  and  OT) . 

Advanced  development  is  conducted  by  one  or  more  contractors.  Personnel 
constraints  determined  in  the  Conceptual  phase  are  a  required  part  of  the  con¬ 
tracts.  The  goal  of  including  these  constraints  is  to  develop  hardware  that 
average  soldiers  can  operate  and  maintain.  The  contractors  are  required  to  pro¬ 
vide  a  Task  and  Skill  Analysis  (TASA)  for  each  operator  and  maintainer. 

The  system  proponents  determine  critical  tasks  from  the  contractor's  TASA. 
The  proponents  also  evaluate  training  and  training  device  requirements  and  esti¬ 
mate  whether  the  operators  and  maintainers  will  require  new  military  occupational 
specialities  (MOS),  new  additional  skill  identifier  (ASI),  or  modification  of 
existing  MOS  or  ASI. 

The  personnel  task  and  skill  evaluation  information  is  used  to  develop  the 
personnel  and  training  factors  and  criteria  for  DT/OT  I,  including  the  skill  re¬ 
quirements,  mental  and  physical  demands,  behaviors,  high  risk  tasks,  needs  for 
training  development  and  training  materials,  and  human  resource  development. 

The  results  of  DT/OT  I  are  used  to  update  the  Outline  Individual  and  Col¬ 
lective  Training  Plan  (OICTP) ,  Provisional  Qualitative  and  Quantitative  Personnel 
Requirements  Information  (PQQPRI),  initial  unit  structures,  and  tentative  Basis 
of  Issue  Plan  (BOIPT) .  Update  of  the  training  support  plan  includes  descrip¬ 
tions  of  training  devices,  training  methods  and  media,  training  extension  courses 
soldiers'  manuals  and  trainer's  guides,  field  manuals,  job  performance  aids,  and 
other  performance  guides. 

The  results  of  DT/OT  I  are  also  analyzed  to  determine  logistic  support 
problems  and  to  assess  total  life  cycle  costs.  The  integrated  logistic  support 
(ILS)  requirements  are  revised  and  alternative  logistic  support  concepts  are 
examined.  The  estimates  of  life  cycle  costs  for  each  alternative  system  and 
the  resource  requirements  of  the  preferred  alternative  are  presented  in  annexes 
to  the  Decision  Coordinating  Paper  (DCP)  prepared  fer  ASARC/DSARC  II. 

One  of  the  documents  that  supports  the  DCP,  the  integrated  program  summary 
contains  manpower  and  training  implications.  The  manpower  implications  cover 
trade-offs  among  hardware  designs,  differences  in  manpower  for  old  and  new  sys-  ■ 
terns,  mar.oower  sources,  new  occupations,  job  task  identification,  and  planned 
test3  and  evaluations  to  verify  the  manpower  estimates.  The  training  implica¬ 
tions  cover  plans  for  reaching  and  sustaining  the  required  operator  and  main¬ 
tainer  proficiency,  plans  for  formal,  on-the-job,  and  unit  training,  number  of 
personnel  to  be  trained,  training  costs,  and  training  requirement  summaries  by 
fiscal  year. 
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The  TRADOC  proponent  develops  the  Required  Operational  Capability  (ROC)  or 
Letter  Requirements  (LR)  that  include  personnel  considerations  believed  to  have 
impact  on  further  devflopraent  of  the  materiel  system  and  personnel  support. 

These  considerations  ..nclude  training  and  training  device  requirements,  personnel 
interface  with  existing  and  proposed  equipment,  system  safety,  and  human  engi¬ 
neering  factors. 

The  Acquisition  ?lan,  prepared  by  DARCOM  in  this  phase,  covers  such  per¬ 
sonnel  and  training  requirements  as  new  skills,  individual  and  crew  training 
requirements,  training  devices,  training  facilities,  and  technical  documentation. 

Personnel  estimates  are  used  to  develop  the  Initial  Recruiting  and  Train¬ 
ing  Plan  which  indicates  personnel  and  training  activities  that  must  precede 
deployment . 

The  DCP  and  other  documents  are  reviewed  in  ASARC/DSARC  II  upon  completing 
advanced  development. 

CTEA  Implications 

Two  CTEA  are  roconnended  during  the  Demonstration  and  Validation  phase, 
both  following  DT/O'f  I  and  before  the  ROC.  The  second  is  a  revision  based  on 
information  gathered  for  the  PQQPRI. 

’  During  DT/OT  I  prototype  systems  are  demonstrated  to  show  that  all  techni¬ 
cal  risks  have  been  minimized  and  that  a  working  prototype  exists.  Prior  to 
DT/OT  I  the  training  program  for  high-risk  tasks  is  used  to  train  the  test 
participants.  If  it  is  not  used  or  is  used  so  that  insufficient  data  can  be 
derived  about  effectiveness  (e.g.,  special  troops  used,  no  control  over  training, 
etc.)  then  this  CTEA  resembles  a  CTEA  in  the  Conceptual  phase.  If  field  data 
do  exist,  however,  the  CTEA  serves  as  an  analysis  of  the  training  programs  to 
verify  estimates  of  trainability  based  on  the  effectiveness  of  the  training. 
Estimates  of  system  effectiveness  degradation  caused  by  training  problems  are 
also  identified . 

The  COEA  in  this  phase  is  designed  to  support  the  ROC  by  assessing  the 
attainability  of  operational  effectiveness  for  the  alternative  system  designs. 
The  CTEA  has  that  function  but  in  addition  diagnoses  and  prescribes  training. 

It  uses  the  lessons  learned  during  DT/OT  I  to  revise  projections  of  training 
programs,  training  device  requirements,  and  the  OICTP. 

The  issues  addressed  at  this  phase  involve  the  high  risk  tasks  and  more 
detail  about  personnel  and  training: 

o  What  are  this  relative  merits  of  the  alternative  system  approaches 
to  training  on  the  high-risk  tasks? 

o  What  is  the  estimated  frequency  of  training  needed  to  maintain 
proficiency  on  the  high-risk  tasks? 

o  What  devices  are  needed  to  support  institutional  and  ’unit  training 
cn  high-risk  tasks? 
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o  What  are  the  system  training  support  requirements  and  associated 
costs? 

o  What  hardware/ training  trade-offs  are  required  to  reduce  training 
difficulties? 

o  What  is  the  impact  on  current  training  and  evaluation  literature? 

o  What  are  the  test  and  analysis  requirements  for  the  engineering 
development  phase? 

Ideally,  input  data  from  DT/OT  I  indicate  operator  and  maintainer  perfor¬ 
mance  on  prototype  systems  after  training  and  include  performance  errors,  prob¬ 
lems  in  the  training  programs,  time  to  train,  and  training  program  costs.  Out¬ 
puts  of  the  CTEA,  given  those  input  data,  include  the  training  program  strengths 
and  weaknesses,  recommendations  for  improvements,  and  an  update  of  the  cost  and 
trainability  estimates. 

The  second  CTEA  recommended  in  the  demonstration  and  validation  phase  fol¬ 
lows  the  PQQPRI  and  precedes  the  ROC.  Personnel  requirements  are  revised  after 
DT/OT  I  and  the  requirements  document  is  staffed  through  appropriate  Army  head¬ 
quarters.  The  CTEA  at  this  point  considers  the  test  results  plus  training  pro¬ 
gram  revisions  made  as  a  result  of  DT/OT  I.  The  ROC  that  it  supports  through 
the  COEA,  is  the  final  statement  of  the  system  capability  the  military  wishes 
to  acquire.  The  COEA  states  the  best  cost  estimate  and  reaffirms  the  force 
effectiveness  to  be  achieved  given  the  DT/OT  I  results. 

The  CTEA  should  provide  the  training  requirements  data  that  form  part  of 
the  ROC.  It  should  address  those  issues  resolved  by  test  data  and  provide  a 
final  assessment  of  the  trainability  of  the  system  concept.  It  should  also 
provide  specific  training  program  goals  to  be  met  in  the  final  system.  The 
goals  include  statements  of  program  elements  (e.g.,  simulator  design  character¬ 
istics,  media)  as  well  as  statements  of  the  desired  levels  of  training  effec¬ 
tiveness  (e.g.,  transfer  of  training,  minimum  acceptable  levels  of  post-training 
skill  retention) . 

In  addition  to  information  in  the  PQQPRI,  data  available  to  be  used  in  the 
CTEA  incude  DT/OT  I  data  and  revisions  to  training  programs.  Outputs  are  train¬ 
ing  requirements  to  ensure  effective  training  (for  the  ROC)  and  assessments  of 
training  program  effectiveness. 

The  second  CTEA  in  the  Demonstration  and  Validation  phase  is  a  direct  up¬ 
date  of  the  preceding  CTEA.  The  analyst  needs  only  to  alter  input  to  the  CTEA 
model  that  represents  revision  in  training  programs  after  DT/OT  I. 

C.  Full-Scale  Engineering  Development  Phase 

During  the  Full-Scale  Engineering  Development  phase  a  system,  with  all 
items  necessary  for  its  support,  is  fully  developed  and  engineened,  fabricated, 
tested  (DT/OT  II),  and  initially  type-classified.  Concurrently,  non-materiel 
aspects  required  to  field  an  integrated  system  are  refined  and  finalized. 

DT  II  evaluates  the  technical  capability  of  the  materiel  and  support  sys¬ 
tems  including  personnel  and  training  requirements.  DT  II  results  are  used  in 
the  decision  to  initiate  full  production. 
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OT  II  tests  the  engineering  prototype  to  estimate  its  operational  effec¬ 
tiveness  and  suitability  in  a  realistic  military  environment.  OT  II  is  designed 
to  employ  personnel  of  the  MQS  required  by  the  system  and  to  test  the  technical 
manuals,  training  devices,  other  training  material,  personnel  requirements,  and 
support  concept.  Instructors  trained  by  the  contractor  train  operators  and  sup¬ 
port  personnel  for  OT  II  using  the  training  programs  developed  for  the  system. 

LCShM  documents  are  revised  using  the  test  results.  The  training  proponent 
determines  the  changes  in  the  Army  Training  and  Evaluation  Program  (AFTEP)  for 
the  new  system.  Also,  the  trainer  drafts  field  manuals  containing  doctrine, 
organizational  material,  and  training  evaluations. 

The  proposed  system,  with  prototypes,  is  analyzed  to  determine  enhancements 
in  reliability,  maintainability,  support  concepts,  and  logistics.  The  logistic 
support  analysis  contains  qualitative  and  quantitative  requirements  for  support 
and  test  equipment,  personnel,  training,  and  facilities.  The  logistic  support 
information  is  issued  in  equipment-oriented  publications  such  as  technical  man¬ 
uals.  Proponent  service  schools  coordinate  logistic  support  analysis  records 
( LSAR )  summary  sheets,  training  and  personnel  doctrine,  draft  equipment  publi¬ 
cations,  and  organizational  structures. 

Documents  revised  on  the  basis  of  DT/OT  II  results  are  the  training  plan, 
the  Basis  of  Issue  Plan  -  Complete  (BOIPC),  and  QQPRI.  Considerations  at  the 
stage  of  the  final  QQPRI  are: 

o  Sufficiency  of  training  to  provide  required  proficiency. 

o  Appropriate  MOS  and  ASI. 

o  Inclusion  of  all  system  components  and  subcomponents  in  the 
QQPRI  documentation. 

o  Adequate  number  of  trained  personnel  to  support  the  system. 

At  Milestone  III  a  draft  decision  coordinating  paper  (DCP)  indicates  the 
status  of  the  system  after  DT/OT  II  and  refinements  of  requirements.  The  DCP 
includes  the  Integrated  Personnel  Support  (IPS)  annex  that  summarizes  the  imple¬ 
mentation  plan  for  the  acauisition  cycle  of  the  system.  It  covers  changes  in 
manpower  estimates  from  Milestone  II,  effects  on  manpower  of  reliability  and 
maintainability  levels,  shortfalls  in  meeting  manpower  requirements,  new  occupa¬ 
tions  not  yet  approved  and  programmed,  and  future  plans  for  evaluating  manpower 
requirements.  The  IPS  also  presents  summaries  of  formal  training  requirements 
by  fiscal  year  and  occupation,  and  plans  for  the  following:  attaining  ar.c  main¬ 
taining  proficiency  required  to  operate  and  maintain  the  system,  additional  re¬ 
sources  required  to  train  the  initial  set  of  personnel,  training  reserve  compo¬ 
nents,  and  validation  of  proficiency  and  performance. 

ASAPC/DSARC  III  authorizes  full-scale  development  or  limited  production  of 
major  systems.  The  item  is  type-classified  as  standard  if  all  critical  issues 
were  resolved  in  development  and  operational  testing.  Limited  production  is 
directed  if  further  testing  is  necessary. 


CTEA  Implications 


The  CTEA  in  the  Full-Scale  Engineering  Development  phase  is  recotmended 
after  DT/OT  II  and  preceding  documentation  for  ASARC/DSARC  III.  This  CTEA 
assesses  training  costs  to  update  earlier  cost  estimates,  this  time  based  on 
experience  with  total  training  requirements.  In  addition,  it  evaluates  the 
match  between  training  requirements  and  performance.  It  is  also  diagnostic 
and  prescriptive  to  improve  week  training  programs. 

The  issues  to  be  addressed  are: 

o  How  effective  was  the  training  program  in  teaching  the  required 
tasks? 

o  Which  tasks  were  not  performed  adequately  during  OT  II  because 
of  poorly  designed  hardware,  poorly  designed  training,  insufficient 
training,  or  improper  personnel  selection? 

o  What  are  the  impacts  of  training  related  deficiencies  on  opera¬ 
tional  effectiveness? 

o  What  additional  training  resources  are  required  to  attain  the 
system  performance  objectives? 

o  What  are  the  requirements  for  additional  training  development 
and  testing  prior  to  full  production? 

Data  input  for  the  CTEA  at  this  stage  are  from  DT/OT  II,  and  the  output 
is  the  CTEA  assessment  of  the  training  package.  This  CTEA  provides  information 
for  the  COEA  conducted  in  preparation  for  ASARC/DSARC  III. 

D.  Production  and  Deployment  Phase 

During  Production  and  Deployment  equipment  is  procured  and  distributed, 
operational  units  are  trained,  and  logistics  support  is  provided.  Acquiring 
and  training  the  personnel  demands  that  the  final  MQS  decision  be  rendered  and 
the  manpower  secured,  the  TOE  be  approved,  personnel  requirements  and  training 
schedule,  new  equipment  training,  including  technical  documentation,  be  com¬ 
pleted,  and  the  ICTP  be  approved.  Training  of  personnel  needs  to  be  started 
in  sufficient  time  to  support  the  new  system  in  the  field.  Resident  training 
is  based  on  revisions  of  training  plans.  It  requires  the  training  equipment, 
devices,  and  ether  training  materials. 

Unit  training  begins  when  the  units  receive  the  system  and  it  must  be  com¬ 
pleted  to  achieve  the  initial  operational  capability.  The  training  is  guided 
by  the  ARTEP  to  include  critical  tasks  for  the  system's  wartime  mission  and  is 
conducted  by  the  cadre  who  received  new  equipment  training.  To  achieve  opera¬ 
tional  reaciness  the  operator  and  maintenance  personnel  in  the  unit  need  to  be 
fully  trained.  When  the  unit  has  the  capability  to  perform  the  mission  the  goal 
of  initial  operational  capability  is  achieved. 

The  Army  LCSJ-W  allows  the  possibility  that  a  Milestone  III  decision  auth¬ 
orizes  limited  production  with  a  follow-on  review  for  full  production  and  de¬ 
ployment.  Thus,  the  Army  has  procedures  to  review  the  system  at  a  later  date, 


ASARC  III A  and  DSARC  III  A.  Reviews  beyond  DSARC  III  require  individual  consid¬ 
eration  and  the  reviews  vary  with  the  situation. 

DT  III  is  conducted  on  initial  production-run  items  to  verify  that  the 
items  comply  with  specifications.  DT  III  determines  if  transition  from  an 
engineering  prototype  to  production  succeeded. 

OT  III  is  conducted  on  initial  production-run  items  to  estimate  their 
operational  suitability.  The  test  covers  tactics,  training,  supportability, 
and  organization. 

The  final  QQPRI  is  prepared  eighteen  months  prior  to  deployment.  It  re¬ 
flects  all  changes  in  design  and  organization  and  the  results  of  the  DT  and  OT. 
It  supports  the  final  M05  decision. 

BOIP  II  describes  the  TOE  organization;  support  equipment  for  the  system, 
where  and  hew  much  will  be  deployed,  and  the  personnel  implications1.  The  BOIP 
is  used  to  justify  the  distribution  of  the  system  until  the  new  TOE  and  IDA 
are  prepared. 

The  acquisition  plan  (AP)  records  program  decisions  and  analyses  of  tech¬ 
nical  options  and  life  cycle  costs  for  development,  testing,  production,  train¬ 
ing  support,  and  logistic  support  of  the  system  incorporating  the  DT  anji  OT 
results . 

The  Decision  Coordinating  Paper  (DCP)  at  the  IIIA  stage  contains  informa¬ 
tion  for  the  optimum  production  quantities  of  the  system  that  are  consistent 
with  readiness  and  modernization.  The  annexes,  including  the  Integrated  Program 
Sunmary  (IPS)  are  prepared  for  the  ASARC  and  DSARC.  This  IPS  contains  all  up¬ 
dated  information  from  tests  and  experience  including  changes  in  manpower  and 
training  requirements .  The  resources  funding  profile  provides  the  costs  of 
operation  and  maintenance,  personnel,  and  support  for  each  alternative. 

CTEA  Implications 

Given  DT/OT  III  data,  the  final  CTEA,  conducted  to  update  estimates,  sup¬ 
plies  information  for  ASARC/DSARC  IIIA. 

E.  Constraints  on  CTEA  Methods 

Constraints  on  CTEA  methods  derive  from  the  location  of  the  CTEA  within 
the  LCSMM,  the  availability  and  quality  of  data,  the  background,  skills,  and 
numbers  of  personnel  available  to  perform  CTEA,  and  the  length  of  the  time- 
frame  within  which  a  CTEA  must  be  performed. 

1 •  CTEA  Location.  The  principal  constraints  imposed  by  virtue  of  CTEA 
location  arise  from  the  absence  of  empirical  effectiveness  data  on  new  system 
tasks  before  DT/OT  I.  Tasks  of  other  (fielded)  systems  may  be  used  to  esti¬ 
mate  the  effectiveness  of  the  training  of  the  tasks  of  the  developing  system, 
but  this  approach  lacks  the  power  to  discriminate  among  training  program  alter¬ 
natives  that  may  be  obtained  through  empirical  effectiveness  data  for  the  new 
system.  Thus,  CTEA  performed  before  DT/OT  I  must  have  methods  for  estimating 
effectiveness.  If  they  are  performed  before  training  program  alternatives  have 
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been  described ,  they  must  also  have  methods  for  predicting  training  program  al¬ 
ternatives.  Since  the  prediction  of  training  program  alternatives  usually  re¬ 
quires  some  sort  of  a  task  list,  the  CTEA  must  have  methods  for  generating  task 
lists  if  none  are  available.  After  DT/OT  I  the  empirical  effectiveness  data 
situation  changes.  Because  the  concern  of  DT/OT  I  is  the  demonstration  of  the 
validity  of  alternative  materiel  ooneepts,  there  will  likely  be  no  training  pro¬ 
gram  and  thus  no  comparative  effectiveness  data.  Thus,  even  after  the  Demon¬ 
stration  and  Validation  phase  has  begun,  the  CTEA  must  have  the  methods  of  the 
earlier  Conceptual  phase  CTEA.  After  DT/OT  II  there  still  may  not  be  enough 
empirical  effectiveness  data  to  answer  questions;  it  may  be  necessary,  for  ex¬ 
ample,  to  consider  a  training  program  alternative  not  predicted  in  advance  of 
DT/OT  II  and  therefore  not  tested.  Thus,  even  after  DT/OT  II  CTEA  may  require 
the  methods  of  the  earlier  concept  phase  CTEA.  Variability  in  CTEA  location  or 
timing  results  from  variability  from  one  acquisition  program  to  another.  Data 
other  than  task  lists,  training  program  alternatives,  and  effectiveness  data 
are  also  available  at  different  phases  and  events  of  the  LCSM. 

2.  Availability  and  Quality  of  Data.  Because  any  particular  acquisition 
program  can  be  expected  to  conform  to  LCSM  only  in  an  approximate  way,  it  can¬ 
not  be  expected  to  consistently  yield  to  CTEA  the  data  implied  by  the  LCSMM. 

Thus,  while  the  LCSM  as  described  by  Rhode  et  ad.  (1930)  yields  Organizational 
and  Operational  Concepts  in  time  to  provide  input  to  a  CTEA  located  to  provide 
input  to  the  CFP,  an  actual  acquisition  program  may  not  yield  such  information 
:il  much  later.  Likewise,  an  actual  acquisition  may  not  even  yield  such 
fundamental  CTEA  input  data  as  task  lists  until  very  late  in  the  acquisition. 

(At  least  one  such  situation  has  occurred.)  When  data  identified  or  implied 
oy  the  LCSM  are  available  at  the  times  indicated  by  the  LCSM,  they  may  not 
be  complete  or  may  otherwise  represent  an  inadequate  quality.  Task  lists,  for 
example,  may  be  only  for  maintenance  tasks,  requiring  the  CTEA  to  have  methods 
for  the  generation  of  operator  task  lists.  Task  lists  may  be  grossly  in  error, 
making  it  necessary  for  the  CTEA  to  proceed  as  if  there  were  no  task  lists  at 
ail. 

The  essential  point  of  constraints  arising  from  the  way  in  which  actual 
act.-.-i  cion  programs  are  implemented  is  that  the  selection  or  development  of 
CTEA  ;hods  is  driven  essentially  by  the  available  input  data. 

1.  Available  Personnel.  In  the  course  of  normal  career  assignments, 
senior  enlisted  personnel ,  warrant  officers,  and  officers  can  all  be  expected 
to  be  tas'--;Ct  with  the  conduct  of  CTEA.  Personnel  available  for  CTEA  must  there¬ 
fore  be  e.  pected  to  vary  a  great  deal  in  background  and  skills.  CTEA  methods 
vary  in  complexity,  required  expertise,  and  difficulty.  Some  are  not  complex 
methods  l  require  considerable  expertise  in  training  development  for  success¬ 
ful  application.  Others  require  less  expertise  but  are  much  more  complex. 

4.  Available  Time.  The  time  available  for  the  conduct  of  a  CTEA  appears 
to  be  on  the  order  of  about  120  days.  The  importance  of  this  constraint  would 
to  some  extent  vary  with  the  number  of  persons  available  to  perform  a  CTEA,  but 
the  usual  CTEA  staff  seems  to  consist  of  about  two  or  three.  Some  CTEA  methods 
may  not  be  consistent  with  such  constraints.  The  use  of  an  analogous  task,  for 
example,  is  not  complex  and  does  not  require  a  great  deal  of  expertise,  but  be¬ 
cause  analogous  tasks  must  be  located  (perhaps  through  an  extensive  search  of 
the  documentation  of  fielded  systems) ,  it  could  require  more  time  than  is  avail¬ 
able.  Methods  available  for  the  conduct  of  CTEA  must  also  recognize  these 
constraints. 
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Beoause  of  the  nature  of  constraints  acting  on  CTEA,  the  process 
methods  that  any  particular  CTEA  must  make  use  of  cannot  be  predicted  on  the 
basis  of  the  location  of  the  CTEA  within  the  LCSht*.  Bather,  methods  must  be 
tentatively  selected  on  the  basis  of  the  situation  which  a  particular  CTEA  en¬ 
counters  with  regard  to  available  data.  Tentative  selections  must  be  further 
constrained  by  considerations  of  available  personnel  and  time  limitations. 

F.  Summary 

To  the  extent  that  training  produces  or  maintains  mission-oriented  skills 
required  system  effectiveness  is  obtained.  Tasks  essential  to  mission  accom¬ 
plishment  with  alternative  designs  or  concepts  vary  in  the  difficulty  with 
which  they  are  trained,  so  overall  operational  effectiveness  also  varies.  An 
accurate  determination  of  operational  effectiveness  must  be  based  on  the  maxi¬ 
mum  proficiency  obtained  with  each  design  and  the  cost  of  training  necessary 
to  achieve  that  proficiency.  The  CTEA  should  provide  that  information.  The 
CTEA  methodology  is  iterative  and  is  updated  as  new  information  becomes  avail¬ 
able.  The  CTEA  results  are  increasingly  detailed  and  accurate  as  the  life 
cycle  of  the  system  progresses. 
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COST  AND  TRAINING  EFFECTIVENESS  ANALYSIS  METHODS 


The  literature  of  training  development  describes  a  number  of  methods  that 
are  either  CTEA  methods  per  se  or  applicable  to  some  aspects  or  processes  of 
CTEA  in  the  LCSM.  While  some  of  these  methods  have  been  developed  for  appli¬ 
cation  to  developing  systems,  others  assume  fielded  systems,  and  none  embraces 
all  of  the  requirements  of  CTEA  in  the  LCSW.  A  purpose  of  this  research  was 
to  organize  the  varieties  of  CTEA  and  related  methods  so  that  the  analytical 
needs  of  each  CTEA  location  in  the  LCSM4  could  be  met.  What  was  envisioned  was 
the  synthesis  of  a  family  of  CTEA  methods  applicable  to  the  LCSW  rather  than 
the  development  of  a  new  method  which  would  meet  all  needs. 

To  establish  a  basis  for  the  synthesis  of  CTEA  methods  suitable  for  the 
various,  stages  of  the  LCSM4,  all  seemingly  relevant  CTEA  methods  described  in 
the  current  literature  have  been  examined.  This  examination  of  existing  methods 
has  attempted  to  answer  three  principal  questions: 

o  To  what  extent  does  the  analysis  described  represent  a  true 
CTEA  method  in  the  sense  that  given  precisely  defined  inputs 
another  analyst  could  derive  precisely  defined  effectiveness 
and  cost  output? 

o  To  what  extent  does  the  analysis,  or  the  purported  CTEA  method, 
embody  processes  usable  within  the  contexts  of  a  variety  of 
CTEA  designs? 

o  For  which  stage  of  the  LCSMti,  if  any,  would  the  purported 
method  or  its  embodied  processes  be  most  suitable? 

Each  method  was  examined  in  light  of  the  needs  of  CTEA  in  the  Aimy  LCSW 
regardless  of  its  purported  purposes  or  established  uses.  The  focus  was  on  the 
details  of  processes  rather  than  the  overall  conceptual  scheme.  The  summaries 
of  methods  thus  developed  established  the  basis  for  the  effort  to  synthesize  a 
family  of  CTEA  methods. 

Since  the  DOD  directives  containing  guidance  on  the  LCSW  were  issued,  many 
studies  have  been  published  documenting  from  various  perspectives  the  costs  and 
effectiveness  of  training  systems  for  developing  and  operational  systems.  They 
have  been  variously  approached  as  cost  effectiveness  analyses  (C/E),  cost  and 
operational  effectiveness  analyses  (COEA),  cost  and  training  effectiveness  analy 
ses  (CTEA),  and  weapon  system  training  effectiveness  analyses  (WSTEA).  These 
assessments  have  been  performed  at  different  stages  of  the  LCSNM,  some  fully 
supported  and  iterated  as  required  and  others  given  considerably  less  time  or 
resources . 

The  summaries  of  CTFA  and  related  methods  that  follow  represent  those  that 
appear  most  applicable  to  the  CTEA  now  being  conducted  as  an  aspect  of  the  Army* 
materiel  system  acquisition  processes  but  by  no  means  do  they  constitute  an  ex¬ 
haustive  catalog  of  methods  that  might  be  applied.  Jorgensen  (1979)  has  identi¬ 
fied  some  innovative  methods  and  techniques  with  possible  useful  applications 
within  training  development  and  CTFA  that  are  not  included  here.  Nor  have  all 
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of  the  methods  and  approaches  summarized  below  been  included  in  the  CTEA  Perfor¬ 
mance  Guide  since  some  will  require  additional  study  and  development  (or  revision) 
to  mSHo  them  useful  and  feasible  as  CTEA  methods. 

Most  of  the  following  methods  and  CTEA  approaches  are  listed  under  commonly 
used  names  and  acronyms.  Formal  bibliographic  information  is  included  within 
parentheses . 

1 .  TEEM  (Jorgensen  and  Hoffer,  Prediction  of  Training  Programs  for  Use  in 

Cost  Training  Effectiveness  Analysis^  i97b) 

The  current  acquisition  policy  requires  effectiveness  comparisons  of  the 
training  alternatives  of  developing  weapon  systems.  In  the  early  stages  of  de¬ 
velopment,  however,  these  systems  have  no  training  programs  to  serve  as  a  basis 
or  comparison.  In  the  case  of  truly  innovative  systems,  furthermore,  there  are 
no  analogous  systems  with  training  programs  that  could  provide  base  cases.  A 
predictive  computerized  model  for  CTEA  use  has  thus  been  developed  by  Jorgensen 
and  Hoffer  (1978).  Their  Training  Efficiency  Estimation  Model  (TEEM),  begins 
with  task  analysis,  proceeds  through  selection  of  training  media  and  methods, 
identifies  information  content  and  structure,  and  generates  a  cost/ effectiveness 
ratio.  Iterations  of  the  model  for  various  training  system  options  produce  cost/ 
effectiveness  ratios  for  comparisons  of  the  options. 

In  TEEM  trade-offs  are  recorded  during  the  generation  of  an  estimated 
training  program.  These  trade-offs  estimate  the  potential  decision  cost  for 
the  chosen  training  program  compared  with  an  ideal  program  with  no  constraints. 
That  is,  an  ideal,  unconstrained  program  is  quantified  and  compared  with  a  simi¬ 
larly  quantified  real-world  (constrained)  training  program.  This  measure  pro¬ 
vides  a  standard  of  comparison  for  several  training  programs  generated  with  the 
same  CTEA  method  and  has  potential  advantages  not  only  for  the  comparison  of 
alternative  programs  for  a  given  weapon  system  but  also  for  comparison  across 
weapon  systems. 

The  measure,  the  efficiency  ratio,  represents  a  value  composed  of  the 
efficiency  score  of  an  estimated  program  with  real  world  constraints  divided  by 
the  efficiency  score  of  an  idealized  program  with  no  constraints.  Outputs  of 
this  method  include  a  cost-analyzed  training  program.  Input  requirements  are 
task  list  plus  sufficient  knowledge  of  the  weapon  system  to  permit  inferences 
about  the  nature  of  stimuli,  responses,  and  feedback. 

The  heart  of  TEEM  is  a  set  of  variables  —  called  a  "metalanguage"  — 
used  to  describe  tasks  to  be  learned  and  the  means  (media  and  methods  and,  per¬ 
haps  in  the  future,  content  and  content  structure)  of  learning  them.  When  each 
variable  in  the  description  of  the  task  is  matched  by  the  corresponding  variable 
in  the  descriptions  of  the  training  program,  an  ideal  training  program  is  identi¬ 
fied.  That  is,  for  each  stimulus,  response,  and  feedback  need  presented  by  the 
tasks  there  is  a  med inn-method  combination  in  the  training  program  to  meet  the 
need.  Because  of  costs  and  other  constraints  such  an  ideal  training  program 
cannot  be  achieved.  Rather,  as  the  result  of  designing  a  training  program  within 
real-world  constraints  something  less  than  an  ideal  program  is  actually  achieved. 
The  difference  between  the  real-world  program  and  the  ideal  is  expressed  by  the 
efficiency  ratio. 


The  metalanguage  includes  three  main  classes  of  variables:  stimulus, 
response,  and- feedback.  In  addition,  there  are  method  (functional  context),  con¬ 
tent,  and  structure  variables.  These  variables  represent  relationships  between 
task  conditions  and  requirements  and  learning  media  and  methods  as  identified 
through  a  thorough  review  of  the  training  literature  (Figure  III- 1 ) .  Matching 
tasks  wi;h  media-method  combinations  through  this  metalanguage  assures  that  the 
training  program  is  responsive  to  research  findings  regarding  the  relationships 
between  tasks  and  media/methods. 

TEEM  begins  with  the  identification  of  task  information  in  the  form  of 
standard  ISD  task  statements  or,  if  the  materiel  system  has  not  progressed  to 
the  stage  that  would  make  these  statements  available,  in  the  form  of  gross  task 
lists  derived  from  materiel  system  descriptions  (draft  TMs,  LSARs,  etc.).  Once 
the  analyst  has  identified  the  tasks,  he/she  describes  the  task  stimulus,  re¬ 
sponse,  and  feedback  variables.  To  describe  the  task  stimulus  variables,  for 
example,  the  analyst  scans  a  list  of  39  stimulus  variables  and  records  those 
that  apply  (e.g.,  audio  cues,  intensity,  channels  or  sources).  To  describe  tne 
response  variables  of  the  task  the  analyst  repeats  the  process  using  the  list  of 
16  response  variables.  Likewise,  the  list  of  17  feedback  variables  is  scanned 
to  describe  the  task  feedback. 

Unless  a  task  is  trained  alone,  it  is  assigned  to  a  group  of  tasks  ac¬ 
cording  to  its  functional  context  (training  method).  Functional  context  vari¬ 
ables  apply  to  groups  of  tasks  that  are  performed  together  on  the  job  and  are 
thus  trained  together.  In  the  case  of  tasks  resulting  from  a  formal  ISD  process, 
the  relationship  of  tasks  to  terminal  learning  objectives  is  given  in  the  form 
of  learning  hierarchies;  in  the  case  of  gross  task  information  it  may  be  possi¬ 
ble  to  estimate  terminal  learning  objectives  from  system  scenarios  and  then  to 
cluster  tasks  accordingly. 

/ 

The  functional  context  (training  method)  is  described  for  clusters  or 
groups  of  tasks  using  13  variables  that  describe  the  context  in  which  a  task 
group  or  cluster  is  performed  and  should  thus  be  trained.  For  example,  the 
task  may  require  individual  performance  (performed  alone).  It  may  be  primarily 
physical  (overt  bodily  actions),  unstable  (not  constant  or  regular  but  subject 
to  continued  change) ,  and  characterized  by  a  physical  context  of  low  impact 
(the  physical  environment  has  little  effect  on  the  performance  of  the  task)  and 
a  psychological  environment  of  low  impact  (the  psychological  environment  has 
little  effect  on  performance  of  the  task). 

The  selection  of  media  (training  devices  and  materials)  is  accomplished 
bv  the  TEEM  computer  program  in  which  training  devices  and  materials  have  been 
described  using  the  same  variables  used  to  describe  the  tasks.  Media  with  the 
highest  nur.ber  of  matches  with  tasks  become  candidates  for  selection. 

The  media  selection  procedure  results  in  stimulus,  response,  and  feed¬ 
back  media  for  each  task.  Since  an  array  of  media  is  not  supportable  in  the 
real  world,  the  media  set  for  each  functional  group  of  tasks  is  reduced  to  a 
supportable  mmber  derived  from  the  matches  between  task  descriptions  and  media 
descriptions.  The  ideal,  unconstrained  training  program  establishes  the  base 
case  against  which  reduced  media  sets  are  compared.  By  definition,  the  effi¬ 
ciency  of  this  media  set  is  1.0.  All  reduced  media  sets  have  an  efficiency  of 
iess  than  1.0. 
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Figure  III-l.  TEEM  Overview 

(from  Jorgensen  and  Hoffer,  1978) 


III-* 


:  ,f*tf  >  Vi  i-.'ri)*-. .. 


« 


The  TEEM  computer  program,  the  mediun  with  the  lowest  number  of  matches 
across  all  tasks  from  the  media  set  in  each  matrix  (that  is,  the  medium  least 
useful  for  stimulus  presentation,  receiving  responses,  and  providing  feedback). 
Iteration  of  the  procedure  measures  the  fit  of  all  tasks  with  all  functions  and 
this  measure  divided  by  the  measure  of  the  ideal,  unconstrained  case  expresses 
the  efficiency  of  the  reduced  media  set.  The  medium  in  each  matrix  with  the 
lowest  number  of  matches  across  all  tasks  is  removed  from  consideration,  and  the 
efficiency  of  the  further  reduced  media  set  is  calculated.  Iterations  of  the 
procedure  continue  until  all  media  have  been  removed. 

Efficiency  ratios  are  plotted  against  the  number  of  the  iteration.  This 
plot  shows  the  iteration  where  the  efficiency  begins  to  drop  off  steeply  and  the 
analyst  uses  it  to  select  a  media  set  that  provides  the  lowest  acceptable  effi¬ 
ciency  (Figure  III-2) . 

The  method  selection  procedure  uses  functional  contexts  and  their  appro¬ 
priate  instruction  methods.  Selection  is  based  on  the  number  of  matches  between 
task  and  training  approach.  The  method  with  the  highest  number  of  matches  with 
functional  context  variables  is  selected  for  each  task  group. 

Costs  are  determined  by  describing  media  and  methods  In  terms  of  37  cost 
variables  and  the  cost  data  are  analyzed  by  the  computer  program.  Outputs  are 
program  costs. 

Once  costs  have  been  obtained,  a  decision  metric,  a  cost/effectiveness 
ratio,  is  obtained  by  dividing  the  cost  by  the  efficiency  value.  The  analyst 
chooses  alternative  methods,  calculates  their  efficiency  values  and  costs,  and 
obtains  their  cost/ effectiveness  ratios.  The  analyst  could  recortrr.end  the  method 
with  the  lowest  cost  effectiveness  ratio  (the  lowest  cc«t  for  at  least  an  accept¬ 
able  efficiency) ,  or  he  might  recommend  a  method  with  a  slightly  higher  cost 
effectiveness  ratio  to  obtain  a  large  increase  in  efficiency  at  a  higher  cost. 

2.  Training  Consonance  Analysis  (Hawley  and  Thomason,  Development  of  an 

Air  Defense  Cost  and  Training  Effectiveness  Analysis  Methodology! 

2  vol . ,  197^) 

Hawley  and  Thomason's  Training  Consonance  Analysis  (TCA)  technique  is 
a  modification  of. ( TEEM  that  compares  training  alternatives  on  the  basis  of  task 
descriptions  and  the- methods  and  media  employed  to  train  the  tasks.  TCA  uses 
the  same  variables  as  TEEM  to  describe  tasks  and  media-method  combinations. 

Unlike  TEEM,  TCA  uses  the  descriptions  to  indicate  how  close  the  media-method 
combinations  come  to- the  task  description;  that  is,  TCA  yields  an  indication  of 
the  consonance  of, the  task  descriptions  and  the  training  programs.  Hawley  and 
Thomason  further  modified  TEEM  by  adding  the  diagnostic  concepts  "training  de¬ 
ficiency,"  "training  excess,"  and  "training  redundancy." 

o  Training  deficiency:  A  variable  in  the  task  description  dees  not 
occur  in  the  training  description. 

0  Training  excess:  A  variable  in  the  training  description  does  not 
occur  in  the  task  description. 
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Figure  III-2.  Plot  of  Efficiency  Ratios  (the  loss  function) 
(from  Jorgensen  and  Hoffer,  1278) 


o  Training  redundancy:  A  variable  in  the  training  description  which 
is  redundant.  This  occurs  only  if  two  or  more  training  media  and/or 
methodologies  are  combined  in  the  training  description.  The  variables 
co  ran  on  to  two  or  more  media/methodologies  are  redundant. 

These  diagnostics  explain  the  differences  in  the  efficiencies  of  alternative 
training  programs.  The  smaller  the  nunber  of  excesses  and  redundancies,  the 
more  efficient  a  training  program  is  in  media  and  methods. 

TCA  describes  each  task  according  to  the  85  psychological  variables  on 
which  TEEM  is  based.  The  task  descriptions  are  coded  on  data  cards  for  computer 
processing.  The  next  step  is  the  identification  of  the  media  and  methods  for 
training  each  task  perhaps  from  a  training  prescription  such  as  TEEM,  TECEP,  and 
TDDA.  Information  in  the  following  four  categories  is  needed: 

o  The  stimulus  mediun  (through  which  information  about  the  task 
is  presented  to  the  student ) . 

o  The  response  mediun  (through  which  the  student  demonstrates 
what  he  has  learned  about  the  task  on  about  any  other  infor¬ 
mation  presented  through  the  stimulus  medium). 

o  The  feedback  mediun  (through  which  the  student  receives  infor¬ 
mation  about  the  correctness  of  his  response) . 

o  The  training  method  (which  describes  the  pattern  of  inter¬ 
actions  among  students,  instruction,  and  media). 

The  TCA  program  analyzes  the  media  and  method  matrices  to  compare  task 
descriptions,  variable  by  variable,  with  the  descriptions  of  media  and  methods. 
When  a  variable  is  present  in  both  the  task  description  and  the  des'  ription  of 
the  medium-method  combination  used  to  train  the  task,  a  training  consonance  is 
scored.  Otherwise,  training  deficiency,  excess,  or  redundancy  is  scored.  The 
training  consonance  ratio  is  given  by  the  total  nunber  of  training  consonances 
scored  for  the  task  over  the  total  nunber  of  variables  in  the  task  description. 

If  36  training  consonances  are  scored  and  39  variables  are  in  the  task  descrip¬ 
tion,  then  the  ratio  for  that  task  is  .923. 

The  TCA  program  aggregates  output  over  groups  of  tasks  and  over  all  tasks. 
The  output,  therefore,  may  be  used  to  evaluate  groups  of  tasks  (which  correspond 
to  modules  of  instruction)  as  well  as  to  evaluate  individual  tasks  or  the  entire 
program.  The  outputs  of  the  TCA  are  summarized  as  follows: 

o  Outputs  are  given  for  each  task,  for  each  group  of  tasks,  and 
for  all  tasks. 

o  Outputs  for  each  task  include  training  consonance  ratios 
training  deficiencies,  training  excesses,  and  training  redun¬ 
dancies  (see  definitions  of  deficiencies,  excesses,  and  re¬ 
dundancies  above) . 

o  Outputs  for  task  groups  and  all  tasks  include  the  total  number 
of  training  consonances,  the  total  number  of  variables  in  task 
descriptions,  and  the  training  consonance  ratio. 
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Training  consonance  ratios  and  diagnostics  provide  a  basis  for  recommen¬ 
dation  of  a  training  program  and  for  improving  the  program. 

3.  CHRT  (Goclowski  et  al . .  Integration  and  Application  of  Human 
Resource  Technologies  inlTeapon  System  Design:  Coordination  of 
Five  Human  Resource  Technologies.  Vol.  1.  March  197#;  Process  for 
the  Coordinated  Application  of  Five  Hunan  Resource  Technologies. 

Vol.  2,  March  1^78;  and  Consolidated  Data  Base  Specification^ 

May,  1978) 


The  Advanced  Systems  Division  of  the  Air  Force  Hunan  Resources  Labora¬ 
tory  combined  five  technologies  in  the  weapon  system  acquisition  process:  main¬ 
tenance  manpower  modeling,  instructional  systems  development,  job  guide  develop¬ 
ment,  system  ownership  costing,  and  human  resources  in  design  trade-offs.  All 
five  are  applied  individually  during  the  materiel  acquisition  process  and  have 
data  requirements  in  common.  Therefore,  one  objective  was  to  integrate.. and  apply 
the  technologies  to  form  a  coordinated  human  resources  technology  (CHRT) . 

A  second  objective  was  to  design  a  consolidated  data  base  /(CDB)  to  sup¬ 
port  CHRT  application.  The  CDB  establishes  a  common  source  of  information  for 
human  resource  technologies  early  in  the  acquisition  process.  A  CDB  is  estab¬ 
lished  for  each  weapon  system  in  the  acquisition  process  and  it  grows  as  the 
system  develops.  Each  of  the  five  coordinated  hunan  resource technologies  draws 
on  the  CDB  for  input,  and  many  of  the  outputs  go  back  into  the  CDB.  Thus,  one 
data  base  contains  information  relevant  to  reliability,  maintainability,  mainte¬ 
nance  manpower,  operations  manpower,  training,  job  guides,  /and  system  ownership 
costs.  It  supports  human  resource  planning  during  the  acquisition  process 
and  continues  to  support  operational  and  logistic  planning/  after  the  system  is 
deployed. 

It  is  useful  to  examine  the  five  human  resource  technologies  separately. 
Maintenance  Manpower  Modeling  (tfiM)  is  a  method  for  estimating  the  manpower  re¬ 
quired  to  maintain  a  new  system  and  the  effects  of  certain!  trade-offs .  Mainte¬ 
nance  action  networks  are  developed  to  describe  necessary  maintenance;  these 
consist  of  sequences  of  maintenance  events  required  to  complete  the  action. 


They  describe  the  probability  of  occurrence,  time  to  complet 
ment  and  personnel  needed  for  each  maintenance  event.  The 
derived  from  the  analysis  of  comparable  existing  systems.  Tt 
networks  are  paired  with  mission  scenarios  to  simulate  the 
system  under  mission  conditions.  The  simulation  predicts  r 
requirements . 
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Instructional  System  Development  (ISD)  within  CHRT  differs^ from  ISD  in 
other  contexts  in  that  it  is  coordinated  with  other  human  resourcexechnologies 
and  draws  its  data  from  a  base  common  to  these  technologies.  It  results  in 
training  concepts  during  the  concept  phase  of  system  acquisition,  a  training 
plan  during  the  validation  phase,  and  a  fully  developed  training  program  during 
the  full-scale  development  phase.  \ 

Job  Guide  Development  (JGD)  results  in  products  that  may  substitute  fpr 
or  reduce  the  need  for  training.  JGD  at  present  appears  to  be  concerned  with  \ 
maintenance  tasks  although  guides  for  operatinonal  tasks  are  consistent  with  the\ 
JGD  concept. 


While  a  large  component  of  Life  Cycle  Costs  (LCC)  is  the  acquisition 
cost  (research  and  development  and  procurement),  a  comparable  component  is  the 
cost  of  operating  and  supporting  a  system  once  it  has  been  procured.  This 
second  component,  system  ownership  cost  (SOC)  includes  human  resource  considera¬ 
tions  such  as  personnel  requirements  for  operations ,  support ,  and  training. 

Every  decision  made  as  a  result  of  NMM,  ISD,  and  JGD  has  an  associated  cost, 
and  it  is  through  SOC  that  such  costs  are  assimilated  into  the  LCC. 

Human  Resources  in  Design  Trade-Offs  (HRDT)  identifies  points  in  systems 
development  where  the  selection  of  alternatives  has  a  large  impact  on  human  re¬ 
sources.  The  HRDT  Design  Option  Decision  Tree  maps  selections  of  systems  alter¬ 
natives  that  have  occurred  or  will  occur.  The  decision  tree  identifies  trade¬ 
offs,  develops  data,  and  studies  to  conduct. 

CHRT  integrates  Jtfld,  ISD,  JGD,  SOC,  and  HRDT.  The  individual  technolo¬ 
gies  retain  their  distinct  characteristics  but  they  interact  through  a  cornron 
pool  of  information,  the  CDB.  A  human  resource  trade-off  identified  by  HRDT, 
for  example,  is  reflected  in  ISD,  JGD,  and  SOC.  Conversely,  an  excessive  cost 
flagged  by  SOC  or  a  support  problem  identified  by  MMM  will  lead  to  a  revision 
in  ISD. 


CHRT  consists  of  four  activities:  (1)  consolidated  data  base  develop¬ 
ment,  (2)  the  integrated  requirements  and  task  analysis,  (3)  ISD/ JGD  product 
development,  and  (4)  the  impact  analysis.  The  CDB  is  the  key  to  the  consolid¬ 
ation  of  the  five  technologies.  The  CDB  consists  of  all  data  that  the  five 
technologies  develop  separately,  but  through  consolidation  it  eliminates  re¬ 
dundancy  and  achieves  efficiency  and  effectiveness  in  the  use  of  data  (Figure 
III-3) • 


The  CDB  begins  with  historical  data  and  assimilates  updated  data  as  they 
are  developed.  It  includes  data  on  a  reference  system  (an  actual  or  conceptual 
systems),  one  or  more  alternative  baseline  systems,  and  current  systems. 

The  integrated  requirements  and  tasks  analysis  (IRTA)  combines  ISD  and 
JGD  analyses  for  maintenance  and  operations  with  maintenance  manpower  modeling 
(MMM).  IRTA  is  predictive  during  the  conceptual  and  validation  phases,  but  it 
becomes  product  oriented  during  the  full-scale  development  phase.  It  determines 
manpower,  ISD,  and  JGD  requirements  as  well  as  the  detailed  task  analyses  needed 
to  develop  ISD  and  JGD  products.  It  is  an  iterative  process,  drawing  heavily  at 
first  on  historical  data  (comparability  analyses,  maintenance  records,  informa¬ 
tion  on  existing  courses,  characteristics  of  available  personnel,  etc.)  and  then 
assimilating  empirical  data  as  the  system  evolves. 

The  ISD  and  JGD  product  development  activity  is  also  iterative.  As  the 
data  become  more  detailed  it  produces  the  training  and  technical  data  (job  guide) 
concepts,  the  ISD  and  JGD  plan,  and  the  ISD  and  JGD  program. 

Impact  analysis,  the  final  CHRT  activity,  is  the  investigation  of  the 
impacts  on  human  resource  costs  of  a  variety  of  system  alternatives.  CHRT 
assigns  human  resource  and  other  systems  ownership  costs  to  system  design, 
maintenance,  operations,  and  support  alternatives  so  that  these  costs  may  be 
fully  considered  during  the  early  and  critical  acquisition  decisions. 
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4.  TECEP  (Braby  et  al. ,  A  Technique  for  Choosing  Cost-Effective 

Instructional  Delivery  Systems.  TAEG  Report  No,  16.  1975) 

Of  all  available  analytical  methods  with  potential  applications  to  CTEA, 
the  Training  Effectiveness,  Cost  Effectiveness  Prediction  (TECEP)  technique  ap¬ 
pears  to  be  the  most  consistent  with  current  ISD  procedures:  TECEP  and  ISD 
employ  the  same  learning  guidelines  and  algorithms .  TECEP  is  used  during  the 
conceptual  phase  for  training  program  development. 

The  TECEP  technique  begins  with  a  list  of  training  objectives,  classi¬ 
fies  those  objectives  according  to  the  type  of  learning ^algorithm  required, 
selects  alternative  media  systems  to  support  those  algorithms,  estimates  the 
cost  of  each  alternative  delivery  system,  and  identifies  a  cost-effective  in¬ 
structional  delivery  system.  The  technique  is  simple  though  its  developers 
caution  that  it  is  intended  for  use  by  experienced  training  system  designers. 

TECEP  is  applied  within  training  systems  development  (Figure  III-4). 

Once  training  objectives  have  been  analytically  derived  TECEP  identifies  and 
costs  instructional  delivery  systems.  The  three-step  procedure  is  given  in 
Figure  III-5,  and  the  reference  materials  needed  to  implement  the  procedure 
are  shown  in  Figure  III-6. 

Each  training  objective  is  matched  with  one  of  twelve  learning  algo¬ 
rithms.  The  matching  is  through  a  comparison  of  the  task  with  the  various 
algorithms  in  terms  of  action  verbs,  behavioral  attributes,  and  examples  of 
objectives  (Figure  III-7).  Once  tasks  and  objectives  have  been  classified 
and  grouped  according  to  the  learning  algorithms  a  table  is  used  to  select 
delivery  systems  for  each  group  of  tasks  and  objectives.  Each  system  is 
then  analyzed  for  cost-effectiveness  comparisons. 

5.  The  EDM/ CARAF  Method  (Cost  and  Training  Effectiveness  Analysis  Handbook 

f*or  Action  Officers,  1§76) 

The  BDM  Service  Company  Combined  Arms  Research  and  Analysis  Facility 
(CARAF)  guide  for  CTEA  outlines  the  process  rather  than  providing  a  procedure  or 
method  whereby  inputs  lead  to  precisely  defined  outputs.  The  guide  suggests  a 
number  of  measures  appropriate  to  the  types  of  training  systems  identified.  It 
also  requires  an  interval  scale  but  does  not  indicate  how  to  quantify  on  an 
interval  scale. 

The  BDM  guide  to  CTEA  describes  eight  procedures  for  each  of  eight  types 
of  training  systems.  The  procedures  are  quite  similar,  however,  and  differ  only 
in  measures  of  training  effectiveness.  The  eight  types  of  training  systems  are: 
large  group  war  games  simulator,  individualized  or  small  group  lessons,  small 
group  tactical  maneuver  and  deployment  game,  hands-on  performance  aid,  large 
weapon  system  practice  firing  adapter,  small  group  combat  engagement  simulator, 
trouble-shooting  trainer  simulator,  and  small  weapon  system  practice  firing 
adapter.  Suggested  measures  of  training  effectiveness  for  large  group  war  games 
simulators  are:  rates  of  movement,  rates  of  attrition,  combat  power  ratios,, 
casualties,  ammunition  usage,  battles  won,  missions  accomplished,  and  others. 
Measures  of  training  effectiveness  are  suggested  for  each  of  the  eight  types 
of  training  systems. 
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Figure  III-4.  Training  System  Development  Model 
(from  TAEG  Report  No.  16) 
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Figure  III-5. 


Process  Flow  in  the  TECE?  Technique 
(from  TAEG  Report  No.  16) 
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Figure  III-6.  Reference  Materials  Supporting  the  TECEP  Process  Flow 
(from  TAEG  Report  Mo.  16) 


CHARACTERISTICS  OF  TRAINING  OBJECTIVES  THAT  CAN  BE 
ACHIEVED  WITH  SPECIFIC  ALGORITHMS 


NAMES  OF 
LEARNING 
ALGORITHMS 


9.  RECALLING 
PROCEDURES. 
POSITIONING 
MOVEMENT 


ACTION 

VEFAS 

BEHAVIORAL  ATTRIBUTES 

EXAMPLES 

'‘.ctivate  V  1 .  Concerns  the  chaining 
Adjust  or  sequencing  of  events. 

Align  V 2.  Includes  both  the  cog* 

Assemble  nitive  and  motor 

Calibrate  aspects  of  equipment 

Disassemble  set-up  and  operating 

Inspect  procedures. 

'Operate  Procedural  check  lists 

Service  are  frequently  used 

as  job  aids. 


1.  Recalling  equipment 
assembly  and  dis¬ 
assembly  procedures. 

V2.  Recalling  the  opera¬ 
tion  and  check-out 
procedures  for  a 
piece  of  equipment 
(cockpit  check  lists). 

\/3.  Following  equipmant 
turn-on  procedures- 
emphasii  on  motor 
behavior. 


Figure  II2-7.  Sample  of  Mat  'hing  Training  Objective  Characteristics 
with  a  Type  c  earning  Algorithm 
(from  TAEG  '  ->rt  No.  16) 


As  an  illustration  of  the  BDM/CABAF  method  for  CTEA,  the  procedure  for 
individualized  snail  group  lesson  delivery  system  is  summarized  below.  Only 
steps  of  some  substance  have  been  included,  and  the  distinctions  among  these 
have  not  necessarily  been  preserved. 

After  precisely  defining  the  training  system  under  investigation,  the 
analyst  chooses  the  standard  measure  of  effectiveness,  which  is  an  aggregate 
of  less  comprehensive  measures  of  training  effectiveness.  In  this  case  such  a 
measure  might  be  a  summary  index  score  of  effectiveness.  Next,  the  analyst 
chooses  measures  with  positive  or  negative  relationships  to  the  standard  measure 
of  effectiveness.  In  the  case  of  the  standard  measure  identified  above,  these 
might  be  measured  achievement,  measured  retention,  and  completion  time.  In  the 
case  of  the  procedure  for  small-group  tactical  maneuver  and  deployment  instruc¬ 
tional  game,  the  standard  measure  of  effectiveness  might  be  a  platoon  leader 
performance  score,  while  the  measure  would  include  missions  accomplished ,  nunber 
of  losses,  and  platoon  leader  evaluations.  For  each  alternative  system  (includ¬ 
ing  a  base  case  system)  data  would  then  be  collected  for  each  measure.  The 
scores  for  each  measure  are'  weighted  and  a  total  weighted  effectiveness  score 
for  the  base  case  and  each  alternative  is  derived  (Table  III— 1 ) .  Using  a  simple 
cost  model,  the  analyst  estimates  the  Life  Cycle  Cost  of  the  base  case  and  each 
alternative  (Table  III-2). 


Table  III-l.  Calculation  of  Total  Weighted  Effectiveness  Scores  (E) 
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Each  training  system  is  compared  with  others  through  a  cost  bene¬ 
fit  analysis  of  the  general  form: 
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Table  III-2. 


Cost  Model 


THIS 


REPLACES 


COST  AND  TRAINING 
EFFECTIVENESS  ANALYSIS 


NAME: 

DATE: 

/  / 

/  / 

PAGE: 

of 

of 

PREPARED  BY; 
ORGANIZATION; 


1.  RAO  COSTS  BASE  CASS*  Af*  Ag* _ A3 


Lesson  Development 

637,000 

11  1 

3,000.000 

Lesson  Revision 

733,000 

1.000,000 

■ 

SUBTOTAL 

1,370,000 

4.000,000 

2.  INVESTMENT  COSTS 

Purchase  of  Lessons 

Purchase  of  Audio-visual  Eqpt. 

SUBTOTAL 

4,5:6,000 

8,000,000 

3.  OPERATING  AND  SUPPORT  COSTS 

Lesson  Distribution 

1,376,000 

4,000,000 

Logistical  Support 

478,000 

1.000,000 

Administrative  Support 

23,642,000 

5,000.000 

Instructional  Support 

10,708.000 

50,473.000 

SUBTOTAL 

36,204,000 

50.473,000 

9,000.000 

4.  TOTAL  LIFE  CYCLE  COSTS 

5.  TOTAL  LIFE  CYCLE  UTILIZATION 

UNITS 

6.  LIFE  CYCLE  COST  PER  UNIT 

42,100,000 

50,473,000 

21,000,000 

266,500 

.266,500 

266.500 

157.97 

139.39 

78.80 

•These  data  are  fur  illustrative  purposes  only.  f  f  f  f 

^Aj  ^A2  ^A3 
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When  the  two  values  are  equal,  the  compared  systems  are  equal  in  cost 
benefits.  Mien  the  first  term  is  the  smaller  value,  the  base  case  is  better 
than  the  alternative.  And  when  the  first  term  is  the  larger  value,  the  alter¬ 
native  is  better  than  the  base  case. 

6 .  MODIA  (Method  of  Designing  Instructional  Alternatives) .  Reported 

in  5  volumes  by  Carpenfcir-HuffWi  et  al.,  1977) 

MODIA* s  developers  explicitly  characterized  it  as  neutral  with  regard 
to  effectiveness  of  training  designs.  It  does,  however,  reveal  the  impacts  of 
design  decisions  and  thus  encourages  designers  to  consider  alternatives.  Con¬ 
siderations  include  implications  of  the  subject  matter  for  training  resources 
and  strategies,  effects  of  student  characteristics  on  learning,  effects  of 
course  management  and  teaching  strategy  on  learning  and  resources,  and  how 
changes  in  one  course  design  element  influences  the  others. 

The  method ‘is  useful  for  comparing  well-developed  training  program 
alternatives.  Because  of  the  level  of  detail  required,  however,  it  does  not 
seem  applicable  early  in  the  LCSW.  Details  required  as  input  include  whether 
the  course  requires  unusually  expensive  or  scarce  resources,  course  content 
sequencing,  maximum  and  minimum  nunbers  of  students,  effects  of  tests  on 
student  progress,  how  each  unit  of  the  course  will  be  taught,  time  required, 
attrition,  and  availability  and  unit  cost  of  resources. 

Major  output  from  MODIA  includes  projection  of  average  and  peak  student 
loads,  average  time  to  graduation  or  attrition,  student  waiting  time,  percent 
of  time  each  resource  is  used,  resources  required,  and  the  start-up,  annual, 
and  five-year  costs. 

MODIA  consists  of  four  elements:  a  description  of  options  for  course 
design;  a  user  interface;  a  resource  utilization  model  (RUM),  and  a  cost  model. 

The  first  describes  the  choices  in  the  design  of  courses,  the  data  required  to 
use  the  systems,  and  guidelines  for  making  design  choices.  The  second  leads  the 
user  step  by  step  through  a  series  of  decisions  that  result  in  a  course  descrip¬ 
tion.  The  third  simulates  operation  of  the  course  to  identify  required  resources. 
The  fourth  estimates  five-year  investment,  and  operating  costs.  The  relationships 
among  the  four  elements  of  MODIA  and  the  user  are  illustrated  in  Figure  III-8. 


Figure  III-8.  Relationships  Among  MODIA  Elements  and  the  User 


III- 16 


Before  attempting  to  make  use  of  the  computer-based  elements  of  MODI  A 
(the  user  interface,  the  resource  utilization  model,  and  the  cost  model),  the 
user  studies  the  optiors  for  course  design  then  gains  access  to  the  user  inter¬ 
face  ( UI ) .  The  UI  poses  a  series  of  questions  that  lead  the  user  through  the 
design  process.  It  asks  where  the  user  wants  to  begin  since  some  of  the  design 
work  might  have  been  completed  during  previous  sessions.  The  UI  begins  the 
questions  relevant  to  the  phase  in  which  the  user  wants  to  begin.  The  eight 
sequential  phases  of  the  UI  are  as  follows: 

o  Select  the  planning  phase. 

o  Describe  objectives  and  tests. 

o  Describe  student  population  and  course  diversification, 
o  Describe  teaching  policy, 
o  Describe  test  details, 
o  Describe  resources . 
o  Describe  resource  constraints. 

o  Define  resource  utilization  model  (RUM)  parameters.  (These 
tell  the  RUM  when  to  stop  the  simulation.) 

The  program  transforms  the  course  design  for  input  to  RUM,  and  RUM 
analyzes  the  performance  of  the  course  by  simulating  the  flow  of  students  through 
it.  Within  the  processes  of  RUM,  for  example,  students  enter  the  course  at  ran¬ 
dom  or  at  prescribed  intervals,  progress  through  a  sequence  of  learning  events, 
use  different  amounts  and  types  of  resources,  may  be  matched  with  learning  events 
on  the  basis  of  their  individual  characteristics,  may  be  temporarily  held  up 
while  they  wait  for  resources  or  for  other  students,  and  may  fail  and  then  have 
to  repeat  all  or  part  of  the  course.  The  user  probably  begins  with  very  brief 
simulations  to  determine  if  bottlenecks  or  other  problems  exist.  As  he/she  works 
out  the  problems,  he/ she  orders  longer  and  longer  simulations  to  verify  that  the 
course  is  indeed  working  as  desired.  Once  satisfied  with  the  operation  of  the 
course,  the  user  assembles  available  cost  data  along  with  RUM  and  UI  outputs  for 
input  to  the  cost  model.  The  cost  model  then  provides  estimates  of  costs. 

7.  DAIS/TRAMOD  Method  (Czuchry  et  al..  Digital  Avionics  Information 
System:  Training  Requirements  Analysis  Model  UserTs  Guide,  1978) 

The  Training  Requirements  Analysis  Model  (TRAMOD)  is  one  of  a  group  of 
computerized  analytical  models  that  make  up  a  life  cycle  cost  model.  It  is 
similar  to  MODIA  in  that  it  is  a  computerized  means  for  training  design,  but 
unlike  MODIA  it  makes  design  decisions  rather  than  simply  revealing  the  impacts 
of  decisions  already  made.  TRAMOD  selects  from  an  input  list  the  tasks  to  be 
trained.  It  generates  a  training  plan  consisting  of  place  of  training  (school 
or  OJT),  method  of  instruction  (simulation,  performance,  lecture,  etc.),  and 
media  (simulator,  mockup,  etc.).  Finally  it  determines  possible  schedules. 

Task  input  includes  values  for  a  nunber  of  parameters:  criticality;  learning 
difficulty;  frequency;  psychomotor  level;  cognitive  level;  and  estimated  time 
required  to  accomplish  training.  In  the  early  stages  of  design  of  new  equip¬ 
ment,  data  values  are  obtained  from  comparable  operational  equipment. 
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TRAMOD  considers  cost  but  not  effectiveness.  Given  adequate  task  data, 
it  would  appear  to  be  a  valuable  CTEA  method  for  comparing  training  alternatives 
resulting  from  various  constraints  (limits  on  training  time,  equipment  shortages, 
etc.).  The  general  process  of  TRAMOD  is  illustrated  by  Figure  111-9. 

The  preparation  of  the  task  data  base  required  by  TRAMOD  is  a  manual 
operation  of  considerable  difficulty.  The  authors  of  the  method  suggest  that 
”...  the  values  assigned  to  the  task  characteristic  parameters  should  be  based 
on  the  judgments  of  engineers  and  technicians  familiar  with  the  equipment  upon 
which  the  tasks  will  be  performed"  (p.  12),  but  implementation  of  such  a  sugges¬ 
tion  seems  to  require  a  change  in  the  way  hardware  manufacturers  currently  per¬ 
form  task  analyses.  The  requirement  for  such  a  change  constrains  successful 
use  of  the  method. 

Once  the  task  data  base  has  been  prepared  and  entered  the  user  interacts 
on-line  with  the  program  by  specifying  constraints  on  the  selection  of  tasks  to 
be  trained,  the  generation  of  the  training  plan,  and  the  generation  of  the  train¬ 
ing  program  (possible  schedules,  types  and  nunbers  of  media  required,  number  of 
classes,  etc.).  The  user  specifies  the  lowest  (threshold)  value  of  a  task  char¬ 
acteristic'  parameter  that  selects  a  task  for  training.  In  the  task  data  base 
each  task  is  evaluated  for  criticality,  learning  difficulty,  frequency,  psycho- 
motor  level,  and  cognitive  level  on  a  scale  of  1  to  5.  By  setting  the  value  for 
each  of  these  five  parameters,  the  user  places  constraints  on  the  selection  of 
tasks.  The  user  then  specifies  which  of  five  algorithms  will  be  used  to  test 
the  values.  One  such  algorithm,  for  example,  requires  each  of  the  character¬ 
istics  of  each  task  to  meet  the  threshold  value  set  for  it  while  another  requires 
the  average  of  the  characteristics  to  meet  the  average  of  threshold  values.  If 
the  user  were  to  set  a  threshold  value  of  3  for  each  of  the  five  characteristics 
(  criticality,  learning  difficulty,  frequency,  psychomotor  level,  and  cognitive 
?  jvel)  and  were  to  choose  the  first  algorithm  described  above  to  make  the  test, 
t  ien  the  program  would  select  for  training  a  task  evaluated  as  criticality  s  5 , 

*  earning  difficulty  =  3,  frequency  =  3,  psychomotor  level  =  4,  and  cognitive 
avel  =  3.  If  the  user  were  to  choose  the  second  algorithm  described  above  both 
t  asks  would  be  selected  for  training  since  the  average  value  of  the  character¬ 
istics  is  at  least  3.  The  user  could,  in  turn,  select  all  five  of  the  algorithms 
to  make  the  test  and  then  compare  the  results. 

TRAMOD  generates  blocks  of  tasks  to  be  trained  in  the  school  or  on  the 
job.  Tasks  that  must  be  trained  together  are  assigned  to  the  same  block. 

Once  satisfied  with  the  set  of  tasks  selected  for  training,  the  user 
proceeds  to  the  generation  of  a  training  plan  by  answering  a  set  of  questions. 

The  first  question  concerns  how  costs  and  training  times  are  to  be  determined. 

The  first  choice  results  in  direct  use  of  the  time  and  cost  data  contained  in 
the  input  data  set  for  each  task;  the  second  choice  results  in  computation  of 
time  and  costs  as  functions  cf  variations  in  the  levels  of  the  value  of  tasks 
characteristics,  with  the  regression  coefficients  being  provided  bv- the  user; 
the  third  choice  results  in  computation  of  time  and  costs  on  the  basis  of  re¬ 
gression  coefficients  stored  in  the  computer  (TRAMOD).  The  second  question  con¬ 
cerns  how  students  are  to  be  assigned  to  school  and  OJT.  The  options  are  termed 
"mixed"  and  "non-mixed."  In  the  "mixed"  option  (students  are  trained  on  some 
task  blocks  in  the  school  and  some  OJT)  TRAMOD  distributes  task  blocks  between 
the  school  and  OJT  on  the  basis  of  minimal  cost  within  time  constraints.  In  the 
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(from  Czuchry,  et  al.,  1978) 


"non -mixed"  option  (student  is  trained  in  the  school  or  OJT  but  not  both)  TRAMOD 
either  calculates  the  split  of  students  between  the  school  and  OJT  or  separately 
tests  school  training  and  OJT  against  given  time  and  cost  constraints,  depending 
on  choices  made  by  the  user. 

The  next  questions  ask  how  training  objectives  are  derived  from  the  task 
data  and  how  media  and  methods  are  assigned  to  training  objectives.  The  user 
can  allow  TRAMOD  to  make  these  decisions  on  the  basis  of  information  stored  in 
the  computer  or  make  certain  alterations  in  the  way  the  matches  are  to  be  made. 
TRAMOD,  for  example,  matches  a  task  group  evaluated  as  psychoraotor  level  =  4  and 
cognitive  level  =  2  with  Training  Objective  3,  Understanding  Principles  and  Re¬ 
lationships.  For  this  training  objective  it  then  assigns  simulation  as  a  method 
and  simulator  as  medium  to  school  training,  and  performance  (hands-on)  training 
as  a  method  and  mock-up  as  media  to  OJT.  The  user  can  decide  to  match  cognitive 
level  =  2  with  Training  Objective  4,  Learning  Procedural  Sequence.  Further,  he 
can  decide  that  the  methods/media  will  be  discussions  with  transparencies  in  the 
school  and  simulation  with  simulators  in  OJT. 

To  generate  the  training  program  the  user  sets  minimum  and  maximum  class 
size  for  school  training,  identifies  the  task  characteristics  that  serve  as  the 
criteria  for  ordering  tasks  for  training,  and  indicate  which  media  are  optimally 
scheduled  because  of  scarcity.  TRAMOD  prints  schedules  showing  the  order  in 
which  tasks  are  to  be  trained,  whether  a  task  is  to  be  trained  in  the  school  or 
in  OJT,  days  required  to  train  each  task,  the  cost  of  training  each  task,  the 
method  for  each  task,  and  the  medium  for  each. 

After  considering  the  results  of  the  decisions  he/ she  has  made,  the  user 
can  rerun  the  program,  alter  his  decisions,  and  compare  the  new  results  with 
former  results. 

8.  TDDA  (Pieper  et  al.,  Training  Developers  Decision  Dialogue  for 

Optimizing  Performance-Based  Training  in  Machine- Ascendant  MOS , 

1978;  and  Tryout  of  a  Training  Developers  Decision  Aid  for  Opti¬ 
mizing  Performance-Based  Training  in  Machine-Ascendant  MOS,  1979) 

The  TDDA  is  similar  to  TECEP  with  one  new  element  added,  Response  Ac¬ 
ceptance  Mechanisms  (i.e.,  ways  of  providing  for  student  responses).  Unlike 
TECEP  it  was  developed  in  both  manual  and  computerized  forms. 

The  four  functional  elements  of  TDDA  are:  task  description;  training 
prescription;  training  hierarchy  and  sequence;  and  training  cost.  Tasks  are 
described  using  specific  action  verbs  and  the  piece  of  equipment  acted  on. 
Training  is  prescribed  as  to:  learning  algorithms;  stimulus  media;  response 
acceptance  mechanisms;  method  of  instruction;  and  learning  setting.  Training 
hierarchy  is  the  result  of  assigning  tasks  to  resident  training,  on-the-job- 
training,  or  no  training,  while  sequence  is  the  specification  of  the  order  in 
which  tasks  are  to  be  trained.  Relative  costs  of  feasible  training  alternatives 
are  established  through  a  cost-rating  technique.  Figure  III-10  is  a  represen¬ 
tation  of  TDDA. 

The  tryout  of  the  task  description  element  indicated  that  the  use  of 
Soldier's  Manuals  may  not  result  in  adequate  task  lists  and  that  the  action 
verb  list  may  not  be  generalizable  to  all  MOSs.  The  user  of  TDDA  examines  the 
MOS  documentation  to  identify  the  actions  performed  by  an  operator  and  searches 
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the  action  verb  list,  a  part  of  which  is  reproduced  below,  to  find  a  verb  that 
accurately  describes  the  action  performed.  The  user  may  find  it  difficult  to 
decide  which  of  several  nearly  synonyn#  ys  verbs  should  be  selected.  The  item 
of  equipment  acted  on  is  added  to  the  task  description. 

Training  prescriptions  use  the  task  action  verb  to  select  one  of  twelve 
algorithms,  which  are  models  of  instruction  adapted  from  TAEG  Report  #16  (Braby 
et  al . .  1975)  and  TAEG  #23  (Aagard  and  Braby,  1976).  They  are  nearly  identical 
to  those  in  TRADOC  Pamphlet  350-30,  Interservice  Procedures  for  Instruction  Sys¬ 
tems  Development.  Appendix  A  in  Pieper  etT  a!u,  19?9,  lists  action  verbs  with 
the  associated  algorithms . 

Having  selected  a  learning  algorithm,  the  user  then  selects  the  stimulus 
medium  by  identifying  the  general  class  of  stimuli  in  the  job  environment  and 
then  further  characterizes  the  stimuli  by  answering  certain  questions.  The 
general  classes  of  stimuli  are  verbal,  audio,  audio-visual,  visual,  and  tactile. 
Several  of  the  questions  answered  to  further  characterize  the  stimuli  are: 

o  Are  the  stimuli  in  color? 

o  Are  the  stimuli  equipment  indicators? 

o  Are  the  audio  stimuli  voice  only? 

o  Are  the  stimuli  visually  distinct  (not  obscured  or  over¬ 
shadowed  by  peripheral  stimuli)? 

Answers  to  these  questions  lead  to  the  selection  of  one  or  more  of  the 
following  media  types:  audio  tape,  books  with  questions,  microfiche  film,  mock- 
up,  movie,  printed  text,  programed  text,  real  equipment,  silent  film  strip, 
silent  slides,  simulator,  sound  film  strip,  sound  slide,  or  television. 

The  TDDA  requires  the  specification  of  the  response  acceptance  mechanism. 
The  general  class  of  the  response  is  first  determined,  and  then  the  class  is 
further  characterized  through  a  list  of  questions.  The  classes  of  responses  are 
equipment  manipulation,  voice,  written,  and  body  movement. 

Questions  asked  to  characterize  the  responses  include: 

o  Is  the  task  a  maintenance  task  (one  that  requires  access  to  the 
interior  of  the  system)? 

o  Are  the  manipulations  discrete  (as  opposed  to  continuous  as  in 
tracking)? 

o  Are  the  control  displays  of  the  equipment  nonlinear? 

The  available  response  acceptance  mechanisms  follow.  These  are  what 
the  student  acts  on  in  training  as  he  learns  a  task:  audio  tape  recorder, 
books  with  questions,  group  instructor,  mockup,  question  set,  real  equipment, 
simulator,  teaching  machine,  tutor,  and  video  recorder. 


The  stvnulus  mediun  and  the  response  acceptance  mechanism  become  the 
basis  for  the  selfcqtion  of  the  method  of  instruction.  A  set  of  methods  compati¬ 
ble  with  both  the  stinnjlus  mediun  and  response  acceptance  mechanism  is  identi¬ 
fied,  and  an  optimal  m&ttiod  (or  methods)  is  selected  by  the  nunber  of  matches 
between  methods  and  several  additional  characterizations  applied  to  both  the 
stimulus  mediun  and  the  response  acceptance  mechanism.  The  additional  charac¬ 
terizations  applied  to  the  stimulus  mediun  are  as  follows: 

X 

o  Pacing  controller  -  what  Controls  the  speed  of  application  for 
the  presentation  mediun  (program,  student,  or  both)? 

o  Stimulus  content  (visual,  verbal,'' or  audio)? 

o  Next  learning  activity  -  what  controls  the  presentations  sequence 
of  items  (program,  instructor,  student)? 

The  additional  characterizations  applied  to  the  response  acceptance 
mechanisms  are: 

o  Pacing  controller  (student,  instructor,  program). 

o  Next  learning  activity  (program,  instruction,  student). 

o  Type  of  evaluation  (individual  vs.  group;  selected  vs.  constructed). 

o  Feedback  (immediacy  -  immediate  vs.  delayed;  source  -  intrinsic 
vs.  extrinsic). 

The  method  is  selected  from  the  following  list:  case  study,  computer- 
aided  instruction,  demonstration,  games,  group  interview,  guided  discussion, 
in-baseket  exercise,  peer  tutor,  programed  instruction,  programned  practical 
exercise,  role  playing,  study  assignment  book,  traditional  classroom, 
traditional  practical  exercise,  and  tutoring. 

If7~fsr~examgle,  the  pacing  controller  of  the  stimulus  medium  is  the 
student,  then  computer^5±ded-jjjstruction ,  programned  practical  exercise,  and 
study  assignment  book  are  possible'methods  because  all  are  compatible  with  this 
characteristic  of  the  stimulus  medium .  The  method,  most  compatible  with  the 
characteristics  of  the  stimulus  medium  and  the  response  acceptance  mechanism  is 
selected . 

On  the  basis  of  three  criteria  one  of  five  learning  settings  is  chosen. 
The  three  criteria  are: 

o  The  nunber  of  students  involved  (a  small  nunber  vs.  5  or  more). 

o  The  nature  of  student  interactions  (individual  vs.  team). 

o  Whether  or  not  equipment  manipulations  are  required. 

Learning  settings  currently  employed  in  TDDA  are  small  group  site,  large 
group  site,  individual  carrel,  small  group  carrel,  and  traditional  classroom. 


Determination  of  the  training  hierarchy  and  sequence  produces  decisions 
about  whether  a  task  is  to  be  taught  in  resident  training,  in  OJT.  or  not  at 
all.  The  decisions  about  the  order  in  which  tasks  are  to  be  taught  are  also 
made  within  this  element. 

The  task  training  matrix,  which  contains  all  relevant  task  data  (sys¬ 
tem  equipment,  support  equipment,  rank  of  task  performer,  task  dependencies, 
etc.),  is  used  to  make  these  decisions.  All  tasks  within  a  particular  cell  of 
the  matrix  are  ordered  hierarchically  according  to  their  interrelationships  or 
dependencies.  Decisions  about  whether  a  task  is  to  be  taught  in  resident  train¬ 
ing,  in  OJT,  or  not  at  all  are  made  on  the  basis  of  six  additional  character¬ 
istics.  These  are: 

o  Percent  members  performing  -  the  relative  numbers  of  persons 
in  the  MOS  who  perform  the  task. 

o  Percent  time  performing  -  the  relative  amount  of  time  each  job 
performer  spends  performing  the  task. 

o  System  readiness  impact  -  the  effect  of  task  performance  on 
the  system's  ability  to  perform  its  mission. 

o  Time  to  application  -  the  period  of  time  after  arrival  on  the 
job  before  the  task  must  be  performed  the  first  time. 

o  Task  learning  time  -  the  number  of  hours  required  to  learn  \ 
the  task  on  the  job  with  the  supervisor  acting  as  tutor.  ^ 

o  Rank  of  task  performer  -  the  rank  of  the  soldier  \rtio  will 
perform  the  task. 

TDDA  training  cost  estimation  is  relative  only,  but  detailed  cost  in¬ 
formation  may  be  developed  after  the  selection  of  the  most  cost-effective  option 
or  options.  A  relative  cost  indicator  is  computed  for  each ’of  the  two  training 
categories  (resident  training  and  OJT).  Values  for  the  two  categories  are  then 
summed  to  yield  a  training  option  cost  indicator.  This  indicator  is  to  form  a 
ratio  that  reveals  the  relative  costs  of  the  training  options. 

The  training  option  cost  indicator  (TOCI)  was  developed  by  classifying 
all  training  methods  and  all  direct  and  indirect  training  costs  to  derive  a 
mean  rating  of  each  method.  Each  of  the  training  methods  was  assigned  to  a  cost 
class  as  follows. 
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Method  Cost  Class  Training  Methods 

1  Conventional 

Demonstration 
Case  study 
Guided  discussion 

2  Peer  tutor 


3  Tutor 

4  Programmed  instruction 

5  Traditional  practical  exercise 

6  Programed  practical  exercise 

7  Computer-assisted  instruction 


All  direct  and  indirect  training  cost  variables  were  assigned  to  one 
of  the  following  classes:  square  footage  (i.e. ,  space  required),  instructor- 
to-student  ratio,  system  equipment,  furnishings,  expendable  supplies,  training 
aid  development,  and  training  materials  development . 

The  various  classes  were  combined  in  the  cost  rating  matrix  illustrated 
in  Figure  III- 11 .  This  matrix  shows  how  costs  vary  as  a  function  of  training 
method  and  provides  both  a  mean  rating  of  each  method  and  a  cost  multiplier. 
Hours  spent  in  training  in  any  method  are  multiplied  by  the  cost  multiplier  to 
give  the  cost  of  training  in  that  method.  The  sum  of  the  method  cost  indicators 
sunmarizes  costs  for  resident  training  and  for  OJT. 
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Figure  III-11.  Representative  TDDA  Cost-Rating  Matrix 
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9.  B-1  SAT  (Sugarman 

Final  Report,  Vol. 

The  B-1  systems  approach  to  training  (SAT)  developed  a  training  program 
for  the  air-crew  of  the  B-1  strategic  bomber.  It  applied  the  techniques  of 
systems  analysis  to  instructional  system  development  to  assure  that  the  entire 
training  system  would  be  considered  within  an  orderly  and  complete  process. 

ISD  application  is  rarely  as  rigorous  as  in  the  B-1  SAT  study. 

SAT  begins  with  analysis  of  missions  into  the  successively  more  detailed 
levels  of  mission  segment,  function,  task,  and  task  element.  The  description 
of  the  task  element  includes  the  actual  behavioral  initiation  cue,  action  verb, 
control,  and  completion  cue.  All  task  elements  are  then  examined  for  commonal¬ 
ities,  and  a  behavioral  objective  is  developed  to  encompass  each  set  of  common 
task  elements  (Figure  III-12).  Behavioral  objectives  include  a  set  of  common 
task  elements  and  information  about  the  behaviors  as  initial  conditions,  concur¬ 
rent  behaviors,  performance  criteria,  enabling  objectives,  crewmen  responsible 
for  the  behavior  and  interactions  among  crewmen.  Figure  III-13  represents  a 
task  element  while  Figure  III- 14  represents  a  behavioral  objective  that  encom¬ 
passes  that  task  element.  The  task  analysis  is  supported  by  a  computer  program, 
called  the  sorting  program,  which  includes  a  controls  and  displays  catalog  and 
an  action  verb  thesaurus. 

The  controls  and  displays  catalog  contains  such  descriptive  information 
as  the  names  (and  synonyms),  locations,  types,  and  subsystems  of  controls  and 
displays,  while  the  action  verb  thesaurus  contains  synonyms  for  action  verbs 
used  to  describe  behaviors.  The  task  analyst  uses  the  sorting  program  to  store 
task  information  and  to  retrieve  necessary  equipment-related  information.  The 
use  of  the  term  "altimeter"  or  "rate  of  climb  indicator"  locates  the  information 
about  the  display  called  altimeter/vertical  velocity  indicator  and  records  the 
correct  term;  the  use  of  the  action  verb  "respond"  causes  the  more  precise  verb 
"reply"  to  be  retrieved  or  stored. 

All  behavioral  objectives,  including  enabling  objectives,  are  arranged 
hierarchically  and  examined  in  light  of  the  qualifications  of  a  probable  enter¬ 
ing  student.  The  tool  supporting  this  determination  was  the  personnel  qualifi¬ 
cations  catalog,  an  example  of  which  is  presented  in  Figure  III-15.  The  figure 
shows,  for  example,  that  an  FB-111  pilot  coming  into  the  B-1  training  program  as 
a  pilot  would  have  been  using  a  pitch  and  roll  trim  control  identical  to  that  of 
the  B-1.  The  pilot  with  such  a  background  is  not  given  training  on  that  control. 

B-1  SAT  identified  training  device  requirements  including  method  and 
media  selection.  Guidance  for  the  identification  of  training  device  require¬ 
ments  was  taken  from  the  Navy's  Training  Effectiveness,  Cost  Effectiveness  Pre¬ 
diction  Technique  (TECEP)  (Braby,  Henry,  and  Morris,  1974),  modified  to  reflect 
the  special  requirements  of  the  B-1  SAT. 

Media/method  selection  is  an  aspect  of  the  structuring  and  scheduling 
of  courses,  tracks,  and  instructional  blocks.  The  tool  employed  to  this  end  is 
the  Training  Resources  Analytic  Model  (TRAM),  a  set  of  computer  programs  that 
examine  proposed  training  system  of  resources,  schedules,  and  costs.  TRAM  is 
similar  to  MODIA  (Carpenter-Hufftoan,  1977),  though  it  may  not  be  as  usable  a3 
MODIA,  at  least  from  the  CTEA  analyst's  point  of  view. 


et  al. .  B-1  Systems  Approach  to  Training. 
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Figure  III-12.  Systems  Approach  to  Training 
(from  Sugarman  et  £l . ,  1975) 


CONSTRUCTION 


OBJECTIVE:  Matt*  *  coordinated  transition  from  climbing 
-  to  cruise  configuration. 

INITIAL  CONDITIONS:  VIATICAL  VtLOCITY:  MOO 
POWER  LIVIL:  100 
AIM  8PIIO:  *1 
ALTITUDI:  M000 
WING  SWEEP:  13 
A-V  TRIMMED  SOA  CLIMB 

CONCURRENT  BEHAVIOR :  HaoOlng  ramaini  conMant 


TIMA.  AT  OESIAID  ALTITUDE:  -SB 
DESIRED  POWER  LEVEL;  M 
DESIRED  AIA  SPEED:  ,P 
OISI AID  ALTITUDE.  30000 
OESIAID  WING  SWEEP:  48 


INITIATION  CUB 

ACTION 

CONTROL  OR  DISPLAY 

COMPLETION  CUE 

ALTIMETER  -  3BB00 

ADJUST 

THROTTLES 

POWER  LEVEL  INO.  -  90 

ALTIMETER  -  3*800 

TRACK 

PITCH  IND. 

CONTROL  STICK 

PITCH  INO.  ■  0 

ALTIMETER  -  30000 

ALTIMETER  •  30000 
POWER  LEVEL  INO.  -  BO 

AO  JUST 

WING  SWEEP  CONTROL 

WING  SWEEP  INO.  -  45 

ALTIMETER  ■  30000 

AOJUST 

TRIM 

PROPRIOCEPTION  -  NEUTRAL 
PRESSURE 

AIR  SPEED  •  .8 

WING  SWEEP  INO.  *  48 

PIAPOAMANCE:  AIA  SPEED  -  .8  (±kti>  at  crulia  altitude. 

“  ALTITUDE  •  30000  (±ft)  from  dcilrad  altitude  at  cruii*  (TIME <  mc). 

SUBJECTIVELY  SMOOTH  VERTICAL  PLIGHT  PATH  (*.g.  REASONABLE  g  FORCES). 
WING  SWEEP  •  45  (±dag.)  after  adjustment  (TIME  <  tael. 

HEAOING  ERROR  •  Odagraai  FROM  DESIRED  HEADING  (±d*g). 


ENABLING  OBJECTIVES: 


CALCULATE 

Nacaaaary  power  tavel  for  cruii*  air  spaed. 

:  altitude,  temperature,  da  tired  true  air  ipaad. 

CALCULATE 

Nacattarv  altitude  to  initiate  power  level  change. 

:  initial  vert,  velocity,  daiirad  altitude,  air¬ 
speed.  desired  airspeed,  A-V  characteristics. 

CALCULATE 

Necessary  altitude  to  initiate  pitch  change. 

:  vert,  velocity  desired  altitude,  airspeed, 

A:V  characteristics. 

COORDINATE 

Throttles  and  control  stick  to  achieve  *  rapid 
transition  having  the  g  forces  within  a  criterion 
without  undershoot  or  overshoot. 

PREOICT 

Necessary  pitch  changes  tor  level-off  et  desired 
altitude  from  the  vertical  acceleration  end  the 
rat*  of  change  in  pitch. 

TRACK 

Pitch  indication  with  control  stick  to  remain  at 
aero  pitch  at  level-off. 

TRACK 

Heading  indication  with  rudders  and  control  stick 
to  remain  at  desired  heading  throughout  maneuver. 

TASK  ELEMENTS: 

8.1. 1.1  OPERATORS:  Pilot  (and  Copilot) 

6.1. 1.2  INTERACTIONS:  OSO  provides  heading  data 

6.1. 1.3  TIME:  Indefinite,  depending 

upon  conditions 

6.1. 1.4  CRITICALITY:  2 

DIFFICULTY:  2 

Figure  III-14.  Behavioral  Objective  Illustrating  a  Maneuver 
(from  Sugarman  et  al_. ,  1975) 
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CLASSIFICATION  OF  INCOMING  ABILITIES 

•  A  -  PRESENTLY  USING  IDENTICAL  EQUIPMENT 

•  8  -  PRESENTLY  USING  EQUIPMENT  WITH  IDENTICAL 

FUNCTION  BUT  DIFFERENT  QFERATION 

•  C  -  PRESENTLY  USING  EQUIPMENT  FOR  SAME  PURPOSE 

BUT  FUNCTIONALLY  DIFFERENT 

•  D  -  NEVER  USED  COMPARABLE  EQUIPMENT 


Figure  III-15.  Example  of  Personnel  Qualification  Matrix 
(adapted  from  Sugarman  et  al.,  1975) 
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B-1  SAT  has  three  determinants  for  the  scheduling  of  instruction:  the 
hierarchy  of  objectives,  the  instructional  context,  and  resource  management. 
The  hierarchy  identifies  dependencies  within  the  content  structure;  that  is, 
what  must  be  learned  so  that  the  next  skill  or  knowledge  may  be  acquired.  The 
instructional  context  refers  to  the  system  or  the  phase  of  flight  with  whioh 
learning  is  concerned.  The  sorting  program  allows  the  analysts  to  search  the 
data  base  to  determine  all  objectives  associated  with  a  particular  system  or 
phasu  of  flight  and  then  to  group  or  not  group  the  objectives  on  such  a  basis 
depending  on  the  outcome  of  tradeoff  analyses,  TRAM  is  the  means  of  conduct¬ 
ing  the  tradeoff  analyses  between  resources  and  contexts.  Grouping  objectives 
within  a  oertain  instructional  context  may  be  indicated  by  the  commonalities 
among  objectives  but  contradicted  by  the  cost  of  using  training  devices  and 
facilities  in  such  a  way. 

Because  students  of  B-1  training  programs  were  to  come  from  different 
sources,  and  would  thus  have  different  entering  skills  and  knowledge,  dif¬ 
ferent  tracks  were  developed  for  each  distinct  source  of  students,  and  were 
examined  by  TRAM  for  total  cost-effectiveness. 

10.  Army  CTEA  Methods  in  Current  Use 

Army  analysts,  such  as  those  in  the  TRADOC  schools  having  proponency 
for  the  developing  weapon  systems,  perform  CTEA  using  a  number  of  techniques. 
These  techniques  were  developed  in  response  to  severe  constraints  on  input 
data.  They  represent  what  is  feasible  for  the  analysts  at  present  and  what 
the  present  research  aims  to  build  upon  and  improve.  Many  of  their  methods 
and  processes  are  pertinent  to  a  general  model  for  CTEA  in  the  LCSM1  and  are 
reviewed  here  to  identify  and  describe  those  processes. 

Some  of  the  current  CTEA  methods  are  applied  to  implemented  systems 
rather  than  systems  still  in  the  acquisition  process,  but  their  methods  and 
processes  are  relevant  to  CTEA  conducted  after  DT/OT  results  are  available. 
Other  of  the  CTEA  apply  to  early  stages  in  the  acquisition  process.  The 
studies  reviewed  were: 

o  DIVAD  Gun  CTEA 

o  Improved  Hawk  (Hawk  PIP)  Training  Development 
o  Roland  Training  Development 
o  Improved  TOW  Vehicle  (ITV)  CTEA 

o  Diagnostic  Rifle  Marksmanship  Simulators  (DRIMS)  CTEA 
o  Bridge  ‘85  CTEA 
o  XM1  Tank  Training  Devices  CTEA 
o  LAW  WSTEA 
o  Ml  6  Hi fie  WSTEA 
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Procedures  in  the  DIVAD  Gun  CTEA  are  applicable  as  early  in  the  LCSM4  as 
the  conceptual  phase,  where  they  are  used  to  generate  task  lists,  and  later  through 
all  stages  until  the  comparison  of  training  program  alternatives.  The  other  CTEA 
applicable  for  generation  of  task  lists  are  the  Improved  Hawk  CTEa  and  Roland 
Training  Development  method.  The  DRIMS  CTEA  contributed  procedures  for  compari¬ 
son  of  training  program  alternatives.  Both  DRIMS  and  the  ITV  CTEA  contain  methods 
for  the  resolution  of  CTEA  issues  in  the  LCSW. 


A.  The  DIVAD  Gun  CTEA.  When  the  DIVAD  gun  system  was  nearing  DT/OT  II 
the  data  available  for  CTEA  corresponded  to  the  conceptual  phase.  There  were  no 
task  data  and  few  other  forms  of  data.  The  CTEA  was  conducted  on  the  basis  of 
knowledge  about  the  class  of  weapon  systems  to  which  the  DIVAD  gun  belongs. 

There  are  certain  tasks  or  types  of  tasks  that  are  common  to  weapons  of  this 
class,  and  there  is  a  great  deal  of  information  about  the  training  of  these 
tasks  from  prior  research.  On  the  basis  of  this  general  information,  and  without 
task  information,  the  CTEA  analyst  drew  upon  prior  documentation  and  subject 
matter  experts  to  formulate  three  training  program  alternatives  and  to  estimate 
the  effectiveness  and  costs  of  the  alternatives.  One  of  the  alternatives  had 
the  highest  effectiveness  estimate  at  lower  cost  and  was  therefore  selected. 

The  procedures  used  in  this  CTEA  represent  a  "generic"  type  based  on 
prior  research,  prior  or  existing  weapons  similar  to  the  developing  system, 
and  subject  matter  expertise. 

B.  Improved  Hawk  (Hawk  PIP)  Training  Development 

One  of  the  processes  of  the  Improved  Hawk  training  development 
effort  corresponds  to  a  potential  CTEA  process,  the  generation  of  a  task  list. 
Improved  Hawk  is  a  modification  of  the  Hawk  design  rather  than  a  redesign,  so 
many  of  the  tasks  of  the  former  are  identical  to  those  of  the  latter,  and  the 
training,  training  requirements,  and  training  implications  are  also  identical. 
Where  tasks  have  been  added  or  changed  new  tasks  were  defined  by  inferring 
tasks  from  the  manufacturer's  draft  technical  manual  (TM).  Figure  III- 16  is 
a  reproduction  of  a  task  analysis  sheet  prepared  by  the  training  developers; 
the  inferred  tasks  are  listed  at  the  left,  and  task  analysis  data  (MOS  to  be 
trained  and  locations  of  training)  are  at  the  right.  A  source  of  this  infor¬ 
mation  is  the  page  of  the  manufacturer's  draft  TM.  Documenting  information 
was  not  always  adequate  so  the  training  developers  worked  through  them  on 
actual  equipment. 


Once  the  new  tasks  were  defined  they  were  used  to  develop  new  train¬ 
ing.  To  the  extent  that  similar  tasks  were  known  to  exist  in  other  systems,  the 
training  of  these  similar  tasks  guided  the  development  of  training  for  the  new 
tasks.  For  other  tasks  they  developed  training  through  standard  ISD  procedures. 


C.  Roland  Training  Development 


The  Roland  task  analysis  procedure  represents  what  a  CTEA  analyst 
might  do  given  adequate  data.  Closely  following  standard  ISD  procedures  and 
using  the  manufacturer's  Logistic  Support  Analysis  Record  (LSAR,  D  Sheets)  and 
draft  TM,  the  task  analyst  developed  a  task  list.  Employing  expert  judgment 
developed  through  the  manufacturer's  training  course  as  well  as  broad  experi¬ 
ence  in  Air  Defense,  he/she  made  certain  revisions  in  the  task  descriptions 
provided  by  the  manufacturer.  He/she  then  analyzed  the  task  list  tc  identify 
critical  tasks  and  to  determine  training  locations  of  tasks. 
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MATRIX  TASK  INVKNTORV 

N.  TAS  CRITICAL  TASK  UST 

MOS 

TRAINING 

LOCATION 

TASK/CATIGORY  N. 

OJT 

S 

Ul 

1 

■  or 

EXTENSION 

joe  AID 

DUTY:  PERFORM  CHECKS  AND  ADJUSTMENTS 

TASKS: 

16 

24 

24 

1 .  Check  T AS  operation! 

E 

E 

G 

X 

X 

X 

16 

24 

24 

2.  Set  up  TAS  locel  control  unit 

E 

E 

G 

X 

X 

X 

16 

24 

3,  Check  TAS  interfece  electronic!  BITE 

E 

E 

X 

X 

X 

16 

24 

24 

4.  Check  end  ediust  TAS  video  conditions 

E 

E 

G 

X 

X 

X 

SUBTASKS: 

16 

24 

24 

4e.  Check/edjuit  vertlcel  roll 

E 

E 

G 

| 

I 

■ 

s 

| 

■ 

1 

s 

16 

24 

24 

1 

I 

I 

1 

1 

4b.Check/edjuit  horiiontel  roll 

E 

E 

G 

I 

8 

1 

I 

16 

24 

24 

I 

I 

■ 

| 

4c.  Check/edjust  numbers 

£ 

E 

G 

1 

I 

1 

I 

■ 

16 

24 

24 

■ 

1 

I 

4d.Check/sdiust  imege  locus 

E 

6 

G 

1 

I 

■ 

16 

24 

24 

I 

1 

■ 

4e.  Check/edjust  mir'or  resolution 

E 

E 

G 

1 

1 

1 

1 

1 

1 

Figure  III-I6.  Reproduction  of  Part  of  Matrix  Task  Inventory  Sheet 
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D.  ITV  CTEA  (TRADOC,  Cost  and  Operational  Effectiveness  Analysis 

for  the  Improved  TOW  Vehicle  (I TV ) ,  Part  IV,  Cost  and  Training 

Effectiveness  Analysis,  June  1976) 

This  CTEA  was  originally  conducted  as  a  part  of  the  COEA  done,  pre¬ 
sumably  during  Phase  III,  Full-Scale  Development,  but  was  subsequently  updated 
to  provide  for  training  development  needs.  It  is  of  interest  methodologically 
because  it  presents  an  examination  of  costs  as  a  function  of  training  location 
(i.e.,  different  Army  training  centers)  rather  than  as  a  function  of  different 
media-method  combinations.  It  assumes  that  the  training  alternatives  are  equal 
in  effectiveness. 

E.  DRIMS  CTEA  (USAIS,  Diagnostic  Rifle  Marksmanship  Simulators 

Cost  and  Training  Effectiveness  Analysis.  August  1976) 

This  CTEA  compared  the  costs  and  effectiveness  of  four  approaches 
to  remedial  rifle  marksmanship  training.  Because  of  the  high  rate  of  failure 
in  rifle  marksmanship  courses,  remediation  is  expensive  and,  conducted  with 
the  Ml 6  rifle  and  ball  ammunition,  possibly  limited  in  effectiveness.  Three 
rifle  marksmanship  simulators  were  considered  as  alternative  approaches  to 
remedial  trailing.  These  were  the  Weaponeer,  the  Lasertrain,  and  the  Caliber 
.22  Rimfire  Adapter  (RFA). 

This  study  is  of  value  to  the  development  of  general  CTEA  methods 
primarily  because  of  the  procedure  followed  to  rank  the  alternatives:  the 
rankings  of  the  alternatives  on  four  separate  factors  were  combined  to  give 
an  overall,  or  final,  ranking. 

This  procedure  is  illustrated  in  Table  III-3.  The  four  ranking  fac¬ 
tors  are  relative  training  effectiveness,  relative  cost,  relative  capability, 
and  relative  user  acceptance.  Rankings  on  each  factor  were  derived  as  follows: 

o  Relative  training  effectiveness  was  given  in  terms  of  the  prob¬ 
ability  of  a  hit  during  record  fire  for  soldiers  given  remedial  training  with 
different  alternatives.  These  data  were  obtained  from  previously  conducted 
studies.  Relative  training  effectiveness  (RTE)  is  defined  as  the  overall  prob¬ 
ability  of  hit  (Py)  associated  with  an  alternative  divided  by  the  probability 
of  hit  associated  with  the  baseline  case,  which  was  the  Ml 6  rifle  firing  ball 
ammunition.  Thus,  the  RTE  of  the  M16  rifle  firing  ball  ammunition  was  1.0, 
while  the  RTE  of  the  Weaponeer,  for  example,  was  1.01  because  the  Py  associated 
with  it  was  slightly  higher  than  the  Py  associated  with  the  Ml  6  firing  ball 
ammunition. 


o  Relative  costs  are  defined  as  the  life-cycle  costs  of  the  alter¬ 
native  over  the  life-cycle  costs  of  the  base  case,  which  is  the  Ml 6  rifle  firing 
ball  ammunition.  Because  the  RFA  can  simulate  25-meter  fire  but  not  record  fire, 
its  .life-cycle  costs  were  compared  only  with  the  25-meter  component  of  Ml  6-ball 
ammunition  life-cycle  costs. 

o  Relative  capability  is  defined,  simply,  as  the  nunber  of  critical v 
skills  taught  by  an  alternative  over  the  total  nunber  cf  critical  skills  of  the 
Ml  6-ball  ammunition  base  case.  The  relative  capability  of  the  Ml  6-ball  ammuni¬ 
tion  base  case  is  1.0,  of  course,  while  the  three  simulators  have  a  somewhat 
lesser  capability  for  teaching  critical  skills. 
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Table  XII-3.  Ranking  Factors  and  Ranking  of  Training  Alternatives 
(from  DRIMS  CTEA,  1978) 


BASELINE  ALTERNATIVES 


RANKING  FACTORS 

M16 

WPNEER 

LTRN 

RFA 

RELATIVE  TRAINING  EFFECTIVENESS  (RTE) 

Record  Fire  P^ 

1.0 

1.01 

0.99 

1.00 

Relative  Training  Effectiveness  Rank 

(1) 

(1) 

(1) 

(1) 

RELATIVE  COST  (RC) 

a.  Table  4,  Section  V 

1.0 

0.08 

0.05 

0.10 

b.  Relative  Cost  Rank 

(4) 

(2) 

(1) 

(3) 

RELATIVE  CAPABILITY  (RCP) 

a.  Table  IV-I 

1.0 

0.86 

0.50 

0.64 

b.  Relative  Capability  Rank 

(1) 

(2) 

(4) 

(3) 

RELATIVE  USER  ACCEPTANCE  (RUA) 

a.  CET  of  Weaponeer  &  Lasertrain 
dated  Oct  77 

1.0 

4.33 

1.0 

0.33 

b.  Relative  User  Acceptance  Rank 

(2) 

(1) 

(2) 

(3) 

TOTAL  OF  RANKINGS  (LOWEST  ix.ORE 

MOST  COST  AND  TRAINING  EFFECTIVE) 

(8) 

(6) 

(8) 

(10) 

FINAL  ALTERNATIVE  RANKINGS 

(2) 

(1) 

(1  (2) 

(3) 
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o  Relative  user  acceptance  is  defined  as  the  number  of  trainers 
who  preferred  an  alternative  training  device  divided  by  the  number  who  preferred 
the  Ml 6  rifle-ball  airmunition  base  case. 

The  rankings  of  the  four  alternatives  of  each  o‘  the  four  factors 
were  summed  to  yield  overall  ranking  scores  such  that  the  lower  the  score  the 
higher  the  overall  ranking. 

F.  Other  Current  CTEA  Methods 


The  remaining  CTEA  methods  reviewed  (Bridge  '85,  XM1  Tank,  LAW, 
and  M16  Rifle)  were  of  general  interest  for  their  processes,  but  did  not  con¬ 
tribute  methods  other  than  those  already  described.  The  LAW  and  Mlb  Rifle 
weapon  system  training  effectiveness  studies  represent  methods  applicable  to 
fielded  systems  which  are  secondary  to  the  present  research. 

1 1 .  Methods  for  the  Analysis  of  Training  Devices/Simulators 

The  process  through  which  training  devices/ simulators  are  selected  and 
developed  for  evolving  materiel  systems  is  complex  and  detailed.  The  entire 
process  does  not  fit  within  the  limited  scope  of  CTEA.  However,  CTEA  may  impact 
on  it  at  several  points:  first,  to  identify  the  need  for  a  training  device; 
later  to  estimate  the  costs  and  effectiveness  of  proposed  training  devices  al¬ 
ternatives;  and  compare  alternatives  with  and  without  training  devices  with 
various  combinations  of  training  devices.  No  special  CTEA  is  necessary  for  the 
identification  of  the  need  for  a  training  device/simulator,  nor  for  the  compari¬ 
son  of  training  program  alternatives  with  varying  levels  of  training  device  use. 
The  estimation  of  the  effectiveness  of  proposed  training  devices  and  simulators 
does  require  a  special  method.  The  prescription  or  development  of  training 
device  and  simulator  characteristics  is  clearly  beyond  the  scope  of  CTEA  prop¬ 
erly  conceived. 

Wheaton  et  al.  (1976)  reviewed  a  number  of  methods  for  estimating  the 
effectiveness  of  proposed  training  devices  and  simulators  and  have  found  them 
to  be  generally  inadequate.  They  proposed  a  new  method  based  on  the  assessment 
of  training  variables  (tasks,  behaviors)  and  device  variables  (appropriateness, 
efficiency,  effectiveness).  It  has  apparently  been  useful  for  estimating  the 
effectiveness  of  proposed  devices,  but  its  validity  remains  to  be  tested. 

The  method  developed  by  Wheaton  et  al.  (1976)  considers  training  device 
effectiveness  as  "a  function  of  the  transfer  potential  of  the  device,  the  learn¬ 
ing  deficit  of  the  trainees,  and  the  extent  to  which  appropriate  training  tech¬ 
niques  are  utilized  in  the  device  (p.  7).”  Transfer  potential  is  given  first  as 
a  function  of  the  commonality  of  tasks  in  the  training  situation  and  the  opera¬ 
tional  setting,  and  second  as  a  function  of  the  similarity  of  the  training  device 
ana  the  operational  equipment.  The  learning  deficit  is  the  difference  between 
entering  proficiency  and  the  proficiency  required  by  the  operational  task.  The 
extent  to  whicn  a  aevice  employs  appropriate  training  techniques  is  determined 
by  an  evaluation  of  each  subtask  trained  by  a  device  in  terms  of  about  one  hun¬ 
dred  principles  and  techniques;  each  subtask  thus  results  in  a  rating  of  the 
device  as  a  potential  means  for  overcoming  the  learning  deficit  represented  by 
the  task.  The  transfer  potential,  learning  deficit,  and  training  techniques 
analyses  are  combined  to  produce  a  device  effectiveness  rating  for  each  subtask, 
and  these  separate  ratings  are  then  combined  to  yield  an  overall  effectiveness 
rating  for  the  device. 
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TRAINVICE  has  been  proceduralized  for  use  by  Army  personnel  with  experi¬ 
ence  in  task  analysis  and  the  development  of  training  objectives.  Five  analyses 
are  necessary  to  apply  the  method: 

o  Task  conmonality  analysis 

o  Physical  similarity  analysis 

o  Functional  similarity  analysis 

o  Learning  deficit  analysis 

o  Training  technique  assessment 


The  procedures  for  each  of  these  analyses  are  given  in  the  manual. 
According  to  the  authors  of  the  method,  these  procedures  represent  an  inter¬ 
mediate  rather  than  an  exhaustive  level  of  detail  and  thus  yield  an  acceptable 
degree  of  precision  while  avoiding  the  extensive  efforts  required  by  other 
methods  for  estimating  the  effectiveness  of  training  devices. 


Training  Device  Requirements  Document  Guide:  A  Procedures  Handbook  for 
Directorate  of  Training  Development  'Project  Offices  for  Devices  (PM  TRADE,  1979) 
includes  the  TRAINVICE  procedures  for  estimating  the  effectiveness  of  devices 
as  an  aspect  of  developing  training  device  requirements.  It  also  includes  pro¬ 
cedures  for  media  analyses  to  be  employed  within  the  process  through  which  an 


expressed  training  device  need  is  analyzed  to  determine  whether  a  training  de¬ 
vice  requirement  should  be  developed  or  the  need  should  be  met  by  alternative 
media.  If  an  expressed  training  device  need  is  justified,  other  procedures  are 
undertaken  to  produce  a  training  device  requirement;  these  include  the  TRAINVICE 


procedures.  Other  procedures  of  the  training  device  requirement  development 
process  are  concerned  with  the  formulation  of  training  device  concepts,  cost 
analysis  of  alternative  training  concepts,  and  validation  of  training  device 


concepts.  In  general,  although  in  less  complex  form,  the  procedures  specify  a 


LCSMM  approach  to  the  development  of  a  training  device  and  thus  include  as  an 


aspect  of  analysis  separate  CTEA. 
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SECTION  IV 


CTEA  METHODS  DEVISED  BY  LITTON 


To  meet  recognized  CTEA  methodological  needs,  Litton  has  revised  or  de¬ 
veloped  three  methods:  an  empirical  method  for  the  prediction  of  training 
programs  and  the  estimation  of  their  effectiveness;  an  interim  method  for 
dealing  with  the  issue  of  trainability  while  more  refined  methods  are  under 
development;  and  a  cost  model. 

A.  The  Analogous  Task  Method  (ATM) 

ATM  takes  advantage  of  information  obtained  in  the  process  of  training 
soldiers  for  fielded  weapons  systems.  This  information  is  brought  to  bear  on 
training  estimation  and  assessment  questions  early  in  the  acquisition  process 
of  a  developing  ^weapons  system.  The  fundamental  link  is  the  identification  of 
a  task  or  tasks  on  fielded  weapons  systems  whose  characteristics  are  comparable 
or  analogous  to  a  task  on  the  developing  system.’  Having  established  the  link, 
the  details  of  training  for  the  analogous  task  (fielded  system)  can  be  used  to 
estimate  the  requirements  for  the  target  task  (developing  system).  Furthermore, 
the  performance  of  soldiers  on  the  analogous  task  and  the  cost  of  training  to 
proficiency  can  be  used  as  estimates  of  corresponding  quantities  for  the  target 
task. 

The  method  is  applied  on  a  task  by  task  basis  with  an  overall  estimate  of 
training  effectiveness  calculated  from  the  task  estimates.  When  discrepancies 
are  found  between  proposed  training  plans  and  training  for  the  analogous  task, 
the  training  plans  are  marked  for  further  investigation.  Likewise  when  the 
assumed  effectiveness  of  training  or  its  cost  differ  markedly  from  measures  of 
analogous  task  training,  training  for  those  tasks  is  also  marked  for  study. 

ATM  method  has  six  steps:  (1)  definition  of  the  critical  tasks  to  be  per¬ 
formed  on  the  developing  system  (the  target  tasks);  (2)  classification  of  the 
target  tasks  to  provide  a  basis  for  finding  the  analogous  tasks;  (3)  analogous 
task  identification  ana  selection;  (4)  assessment  of  training  for  the  analogous 
tasks;  (5)  generation  of  estimates  of  training  for  the  target  tasks;  and  (6) 
aggregation  of  the  effectiveness  and  cost  measures  across  all  tasks  to  obtain  a 
picture  of  training  for  the  developing  system  as  a  whole. 

1 .  Step  1 :  Task  Definition 

The  basic  data  of  the  empirical  method  are  the  description  of  those 
operator  and  maintenance  tasks  essential  to  weapons  system  operation  and  mis¬ 
sion  accomplishment.  This  list  is  compiled  initially  either  by  the  manufacturer 
or  proponent  school  early  in  the  conceptual  phase  of  the  LCSW.  Without  the 
list  of  tasks,  the  analysis  cannot  proceed.  The  analyst  must  either  wait  for 
the  task  list  to  be  produced  or  attempt  to  compile  one.  A  critical  task  list 
may  be  assembled  by  the  analyst  from  a  step  by  step  description  of  system  oper¬ 
ation  obtained  from  the  manufacturer's  technical  personnel.  Where  the  concept 
is  sufficiently  advanced,  construction  of  a  mock-up  aids  the  exact  description 
of  the  tasks.  When  the  Logistical  Support  Analysis  Record  (LSAR)  is  available, 
the  task  descriptions  on  Sheet  D  (Figure  IV- 1)  and  the  Failure  Analysis  on 
Sheet  B  (Figure  IV-2)  can  be  used  to  verify  critical  tasks. 
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Figure  IV-1.  Logistical  Support  Analysis  Record  Sheet 
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The  verbs  that  describe  the  actions  performed  by  the  soldiers  must 
specify  the  nature  of  the  action.  These  verbs  will  be  used  in  the  following 
step  as  a  basis  for  classifying  the  task.  Verbs  like  '’perform**,  "accomplish", 
"continue",  or  "operate"  are  too  general  to  allow  proper  classification.  As  a 
guide  to  the  appropriate  level  of  specificity,  the  analyst  should  compare  the 
verbs  in  his  list  with  those  in  the  task  verb  list  (acceptable)  and  the  general 
verb  list  (not  acceptable).  Those  description  whose  verbs  are  too  general 
should  be  rewritten  with  more  specific  verbs.  It  is  likely  that,  on  rewriting, 
the' "task"  will  prove  to  be  a  group  of  tasks,  each  of  which  should  be  considered 
individually.  The  task  verb  list  is  presented  in  the  CTEA  Performance  Guide. 

2.  Step  2:  Task  Classification 

Hie  target  task  is  classified  by  the  action  verb  in  the  task  descrip¬ 
tion,  the  body  of  knowledge  the  soldier  must  understand  as  general  background 
to  task  performance,  the  stimuli  provided  by  the  equipment  to  initiate  the  task, 
the  response  made  to  that  stimulus,  and  the  feedback  provided  by  the  system  to 
indicate  proper  or  improper  performance  of  the  task.  The  task  type  is  determined 
by  the  action  verb  and  the  knowledge  context  for  each  task.  The  action  verb  is 
compared  with  those  in  the  task  verb  list.  The  list  gives  the  task  type  to  which 
each  verb  meaning  has  been  assigned.  If  the  verb  does  not  appear  in  the  task 
verb  list,  then  the  verb  should  be  compared  with  the  descriptions  and  examples 
of  each  type  which  appear  in  the  synonym  list  The  verb  may  be  an  unlisted  syno¬ 
nym  for  one  of  the  verbs  in  the  synonym  list.  If  not,  the  analyst  should  assign 
the  verb  to  that  category  which  most  closely  describes  it.  The  synonym  list 
appears  in  Annex  C,  Appendix  2  of  the  CTEA  Performance  Guide. 

The  task  categories,  in  some  cases,  include  widely  differing  tasks 
which  cannot  be  distinguised  solely  on  the  basis  of  the  verb.  One  basis  for 
distinguishing  subcategories  is  the  knowledge  context  of  the  task.  For  example, 
computer  tasks  can  be  distinguished  from  electronics  tasks  or  mechanical  tasks 
even  though  they  all  involve  following  procedures.  (A  list  of  knowledge  contexts 
is  given  at  the  end  of  the  synonym  l:st.)  If  the  appropriate  context  does  not 
appear,  the  task  type  should  be  used  without  an  associated  context. 

The  determination  of  stimulus,  response,  and  feedback  types  follows  the 
classification  of  Jorgensen  and  Hoffer  (1978).  The  analyst  examines  the  task 
description  and  determines  which  stimulus,  response,  and  feedback  categories 
best  fit  the  task.  The  stimuli  are  characterized  as  to  modality  (visual,  audi¬ 
tory,  tactile,  etc.),  nature  over  time  (static  or  dynamic;  frequency  of  change), 
number,  and  organization  (formatted  or  not;  simple  or  complex  patterns).  Re¬ 
sponses  are  characterized  by  mode  of  implementation  (verbal,  written,  manipu¬ 
lating,  etc.),  nature  over  time  (static  or  dynamic;  frequency  of  change),  number, 
and  organization.  Feedback  is  characterized  by  modality  (visual,  aural,  written, 
etc.),  time  relationship  to  the  preceding  response  and  the  following  stimulus, 
regularity,  and  frequency. 

3.  Step  3:  Analogous  Task  Identification 

As  more  CTEA  are  performed  the  resulting  data  may  be  stored  in  a 
central  training  data  base  to  facilitate  future  identification  of  analogous 
tasks.  In  the  short  term,  the  analyst  must  identify  the  analogous  task  intui¬ 
tively.  The  first  place  to  look  for  an  analogous  task  is  the  previous  genera¬ 
tion  weapons  systems  being  replaced.  If  the  weapon  system  is  composed  of  major 
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subsystems  (e.g.,  radar  tracking,  missile  launcher),  then  a  comparable  subsys¬ 
tem  on  another  weapons  system  may  provide  the  analogous  tasks  needed.  If 
comparable  subsystems  cannot  be  found,  the  analyst  must  look  for  individual 
tasks. 


To  be  an  analogous  task,  the  task  on  the  fielded  weapons  system  must 
be  in  the  same  task  type,  have  the  same  knowledge  context  (if  any),  and  have 
the  same  response  type.  In  addition,  the  analogous  task  must  have  lesson  plans 
or  materials  which  describe  the  training  for  it  in  detail.  If  an  analogous  task 
cannot  be  found,  then  an  analytical  procedure  (e.g.,  TEEM  or  TECEP)  must  be  used 
to  identify  necessary  training  elements.  The  assessment  measures  obtained  from 
these  methods  do  not  directly  estimate  or  predict  the  performance  of  soldiers 
resulting  from  a  training  program. 

If  more  than  one  potential  analogous  task  is  identified,  the  analyst 
should  attempt  to  choose  among  them.  The  choice. is  determined  by  the  remaining 
task  information  (i.e.,  other  than  the  minimum  criteria):  the  specific  task 
verb,  the  specific  response,  the  stimulus  type,  the  necessary  conditions. 

These  criteria  should  be  considered  in  the  order  given.  Only  tasks  identical 
on  a  prior  criterion  should  be  evaluated  on  a  later  one. 

4.  Step  4:  Training  Assessment  for  the  Analogous  Task 

For  each  analogous  task  identified,  the  analyst  must  obtain  the  details 
of  training.  These  details  will  be  contained  in  the  training  programs  and  mate¬ 
rials  and  the  specifications  for  training  devices  used  in  conjunction  with 
training. 


The  analyst  must  also  obtain  performance  information  to  be  used  as  a 
measure  of  the  effectiveness  of  training.  The  measure  is  defined  as  the  propor¬ 
tion  of  soldiers  achieving  criterion  performance  on  the  task  following  training. 

If  data  from  the  hands-on  portion  of  the  Skill  Qualification  Test  (SQT)  are 
available  for  that  task,  then  they  should  be  used  to  calculate  effectiveness. 

If  not,  the  written  portion  should  be  used  in  their  place.  Where  no  SQT  data 
are  available,  data  from  an  end-of-course  test  should  be  used.  In  the  absence 
of  all  formal  assessment  tests,  the  instructor  should  be  asked  to  estimate  the 
proportion  of  soliders  who  learn  the  task. 

5.  Step  5:  Training  Assessment  for  the  Target  Task 

The  training  on  the  analogous  task  can  be  used  as  the  training  plan 
for  the  target  task  if  no  plan  has  been  formulated.  If  there  is  a  training 
plan,  then  differences  between  the  training  plan  and  analogous  task  training 
will  indicate  potential  difficulties  in  training  to  be  watched  for  in  opera¬ 
tional  testing. 

If  the  training  for  the  target  task  is  identical  to  that  cf  the  analogous 
task,  the  effectiveness  measure  for  the  analogous  task  becomes  the  estimate  of 
effectiveness  for  target  task. 

If  the  training  differs,  the  effectiveness  in  Step  4  becomes  a  criterion 
against  which  the  proposed  effectiveness  is  compared.  Care  must  be  taken  that 
the  criterion  dimensions  for  the  two  ar’e  the  same.  If  more  than  one  analogous 
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task  has  been  identified,  the  data  can  be  combined  in  the  following  manner.  If 
the  training  is  roughly  the  same,  the  effectiveness  measures  should  be  averaged 
to  obtain  the  estimate  of  effectiveness  for  the  target  task.  If  the  training 
differs  among  the  analogous  tasks,  that  with  the  highest  effectiveness  measure 
should  be  chosen. 

As  data  on  training  are  obtained  from  OT  I  and  OT  II,  the  direct  esti¬ 
mates  for  performance  on  the  task  should  be  substituted  for  the  measures  based 
on  analogous  tasks. 

Since  training  for  different  tasks  may  develop  at  different  rates  or 
analogous  tasks  may  be  found  for  only  some  of  the  critical  tasks,  it  is  entirely 
possible  that  some  of  the  tasks  will  have  only  analytical  measures  while  others 
have  performance  measures.  Therefore,  combining  measures  into  a  single  aggre¬ 
gate  effectiveness  measure  will  not  be  meaningful.  Even  if  all  measures  are 
performance  measures,  a  reasonable  set  of  weights  for  combining  the  measures 
may  be  difficult  to  deduce.  In  both  cases,  a  verbal  rather  than  numerical  over¬ 
view  of  training  is  suggested. 

The  method  is  illustrated  in  Figure  IV-3.  Each  rectangle  represents 
a  step  of  the  process,  while  the  processes  indicated  in  the  circles  show  the 
relationship  of  the  process  to  other  processes  of  a  general  CTEA  model  to  be 
described  in  Section  V  (Figure  V-1). 

B.  Trsinability  Analysis 

Once  tasks  have  been  identified  and  alternative  means  of  training  those 
tasks  have  been  predicted  or  developed,  it  is  necessary  to  determine  that, 
given  the  characteristics  of  the  personnel  who  will  man  the  system,  the  tasks 
can  be  trained  to  required  levels  of  proficiency.  The  purpose  of  trainability 
analysis,  thus,  is  to  examine  the  interactions  among  tasks  (especially  stan¬ 
dards)  ,  training  program  alternatives,  and  system  personnel  characteristics. 

The  review  of  the  literature,  however,  has  revealed  no  explicit  method  for 
trainability  analysis.  According  to  a  TRADOC  official  (Telephone  Interview, 

26  June  1980),  however,  such  methods  arc  now  under  development  and  will  prob¬ 
ably  be  available  at  some  time  in  the  future.  This  official  agreed  that  it 
would  be  reasonable  to  apply  the  following  stopgap  method  in  the  interim. 

The  objective  of  this  method  is  to  identify  tasks  that  are  of  great  or 
moderate  concern  as  regards  the  trainability  issue.  Once  those  tasks  have  been 
identified,  certain  trade-offs  may  be  considered  and  recommended:  the  revision 
or  reconsideration  of  certain  training  program  alternatives;  the  alteration  of 
system  concepts  or  designs;  and  the  revision  or  establishment  of  personnel  selec¬ 
tion  criteria. 

The  risk  that  a  task  cannot  be  adequately  trained  given  an  existing  or 
predicted  training  program  and  system  personnel  characteristics  is  judged  to  be 
high,  moderate,  or  low.  Each  system  task  is  thus  evaluated  against  each  train¬ 
ing  program  alternative.  The  criticality  of  each  task  is  also  judged  to  be  high, 
moderate,  or  low.  These  twin  evaluations  are  then  used  to  form  a  trainability 
analysis  matrix  for  each  task  as  trained  by  each  training  program  alternative. 

The  matrix  is  illustrated  in  Figure  IV-4.  The  cells  indicate  the  level  of  the 
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Figure  IV-3.  Analogous  Task  Method 


analyst's  concern.  Where  concern  is  great  or  moderate,  the  analyst  must  resolve 
it  by  conducting  trade-off  analyses  (above)  or  by  calling  attention  to  the  con¬ 
cern  and  recommending  additional  studies. 
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Figure  IV-4.  TASK  TRAIN ABILITY  ANALYSIS  -MATRIX 


C.  The  Cost  Process  Model 

Because  resources  are  limited  and  costly,  choices  among  alternative  systems 
or  programs  must  be  based  on  the  allocation  of  these  resources  among  competing 
demands.  For  example,  if  more  than  one  alternative  satisfies  the  mission- 
requirement  (equal  effectiveness),  the  least  costly  alternative  is  preferred. 
Obviously,  the  cost  analyses,  whether  it  be  absolute  or  incremental,  in  a  CTEA 
is  essential. 

A  cost  model  orders  and  simulates  the  many  actions  involving  the  expendi¬ 
tures  of  resources  (including  time)  for,  in  this  case,  A '-’my  training.  The  cost 
model  allows  the  analyst  to  predict  or  recapture  in  a  systematic  manner  those 
expenditures  of  resources  that  were  or  will  be  germane.  The  first  requirement 
of  a  model  is  that  it  be  relevant  to  the  need,  i.e.,  be  able  to  predict  or  cap¬ 
ture  the  costs  of  training  Army  tasks  or  systems.  Army  training  takes  place  in 
the  institution  (schools)  and  in  the  units.  Most  of  the  institutions  have  base 
or  classroom  instruction  and  instruction  in  the  field.  Institutional  base 
training  can  be  self-paced  or  instructor  oriented.  There  is  individual  training 
and  collective  training  in  the  Army.  An  Army  training  cost  model  should  be  Amy 
j  oriented  and  have  the  capability  of  estimating  training  costs  that  consider  all 

pertinent  variables . 

During  the  literature  review,  Litton  reviewed  many  cost  models  and  approaches 
to  costing  of  training,  eleven  of  which  have  been  abstracted  and  appear  in  Ap¬ 
pendix  C.  Litton  sought  a  model  that  would  provide  the  capabilities  required 
rather  than  to  design  a  new  one.  None  met  all  the  requirements.  For  example, 
the  Army  Life  Cycle  Cost  Matrix,  although  Army  oriented,  is  better  structured 
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for  and  focused  on  budgetary,  finance,  and  accounting  purposes  rather  than  CTEA. 
The  TECEP  model  is  focused  on  institutional  base  training  but  could  be  modified 
to  meet  the  other  requirements.  However,  complete  modification  of  the  TECEP 
model  results  in  the  Litton  proposed  cost  model. 

The  CTEA  analyst  must  retain  a  high  degree  of  flexibility  in  relating  the 
form  of  the  data  available  with  the  equations  herein.  For  example,  where  the 
equations  contain  variables  on  a  "per  student"  basis,  these  variables  are  usually 
then  multiplied  by  the  average  nunber  of  students  to  obtain  total  costs.  If, 
however,  total  cost  data  are  already  available,  then  these  should  be  entered 
directly  as  the  product  of  both  of  the  variables. 

The  Litton  cost  model  is  designed  to  capture  the  significant  costs  of  train¬ 
ing  a  task  or  an  entire  program.  If  cost  elements  are  available  on  a  training 
program  basis,  then  the  model  output  will  be  the  total  cost  estimate  of  the  train¬ 
ing  program.  More  likely,  the  CTEA  analyst  will  have  to  deal  with  analogous  tasks 
and  will  have  to  iterate  the  model  for  each  such  task.  The  grand  total  of  the 
costs  of  training  all  the  tasks,  of  course,  will  give  a  cost  estimate  of  the 
training  program. 

The  Litton  cost  model  is  intended  to  aid  the  CTEA  analyst  to  prepare  recom¬ 
mendations  to  the  decision  maker  regarding  choices  among  alternatives.  It  is 
not  intended  for  budgetary  purposes.  The  Litton  .aodel,  however,  captures  per¬ 
sonnel  and  unit  costs  in  order  to  convert  training  time  and  differences  in  grades 
of  personnel  involved  into  a  measure  compatible  with  the  other  factors  in  the 
decision  process:  dollars. 

It  is  necessary  for  the  CTEA  analyst  to  have  some  understanding  of  the  Five 
Year  Defense  Plan  (FYDP)  and  its  associated  budget  programs  to  assist  in  data 
collection.  Figure  IV-5  shows  the  ten  Army  programs  currently  in  use. 

Represented  by  the  programs  and  subprograms  are  resources:  people,  money, 
materiel,  and  facilities.  It  would  appear  that  all  people,  money,  materiel,  and 
facilities  required  for  training  would  be  provided  under  Program  fiT,  training. 
This,  however,  is  net  the  case.  Program  8T  carries  resources  for  almost  all 
institutional  training  but  training  resources  are  found  also  in  Programs  2,  3, 

5,  6,  7M  and  8M.  Program  2,  for  example,  carries  training  resources  for  the 
individual  and  the  collective  training  of  personnel  in  General  Purpose  Forces. 

On  the  other  hand,  such  activities  as  training  developments,  combat  developments 
and  direct  assistance  to  unit  training  by  institutions  are  not  technically  insti¬ 
tutional  training  but  are  accounted  for  in  Program  8T.  Institutional  training 
of  individual  medical  skills  is  accounted  for  in  Program  SM. 

The  CTEA  analyse  must  also  understand  where  various  levels  and  types  of 
training  are  trained.  Figure  IV -6  is  a  matrix  which  shows  this.  (It  should  be 
noted  that  the  terra  One  Station  Unit  Training  (QSUT)  is  not  unit  training  but  re¬ 
fers  to  the  institutional  training  program  that  combines  aspects  of  basic  training 
(BT>  and  advanced  individual  training  (AIT)  while  the  trainee  is  assigned  to  one 
training  unit.)  The  matrix  portrays  the  usual  situation.  There  are  rare  excep¬ 
tions  where,  for  a  special  system,  some  crews/ teams  will  be  given  collective 
training  in  the  institution. 
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The  Litton  cost  model  concentrates  on  the  initial  individual  and  collective 
training  required  to  attain  proficiency  on  the  system  and  operational  capability. 
Costs  attributable  to  refresher  training  are  omitted.  Proficiency  is  assumed 
when  the  soldier  successfully  completes  the  appropriate  SOT  and  tne  crew/ team 
successfully  completes  the  appropriate  section  of  the  Army  Training  and  Evalua¬ 
tion  Program  (ARTEP). 

The  costs  of  training  can  be  either  direct  or  indirect.  Direct  costs  asso¬ 
ciated  with  institutional  training  consist  generally  of  the  following: 

1.  Pay  and  allowances  of  instructors  and  staff. 

2.  Cost  of  support  rendered  to  the  institution  (including  pay  and 
allowances  of  personnel)  by  TOE  units  whose  mission  is  to 
support  the  institution  (Program  8T). 

3.  Student  pay  and  allowances. 

4.  Student  travel  pay  to  the  institution. 

5.  Cost  of  ammunition  expended  for  institutional  training. 

6.  Student  per  diem  entitlements  at  the  institution. 

7.  Depreciation  of  equipment  dedicated  to  institutional  training. 

8.  Cost  of  consumable  supplies  and  material. 

9.  Cost  of  contractual  services. 

10.  Institutional  overhead  costs  such  as  the  pay  and  allowances  of 
personnel  in  offices  of  the  Conmandant,  Security,  Director  of 
Logistics  and  of  School  Training  Brigades. 

Indirect  costs  of  institutional  training  are  those  costs  not  directly  and 
wholly  attributable  to  training  but  all,  or  at  least  part,  of  which  should  be 
charged  to  training.  These  costs  include: 

1 .  Cost  of  support  rendered  to  the  institution  by  Program  2  TOE 
units.  Military  pay,  operations  and  maintenance,  and  TOE 
equipment  depreciation  are  apportioned  according  to  the  man- 
days  of  support  rendered.  It  is  recognized  that  these  units 
are  in  the  force  structure  to  meet  Joint  Chiefs  of  Staff  re¬ 
quirements  and  that  pay  and  allowances,  for  example,  would  be 
a  cost  whether  or  not  they  support  institutional  training. 

However,  time  diverted  to  support  institutional  training 
should  be  accounted  for  -  especially  when  comparing  systems. 

This  may  be  done  by  converting  this  time  to  dollars.  Addi¬ 
tionally,  using  pay  and  allowances  to  do  this  permits  con¬ 
sideration  of  the  different  grades  of  personnel  required  by 
competing  systems. 
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2.  Proportionate  share  of  the  costs  of  base  operations,  family 
housing  administration,  base  comnunications,  base  medical 
support. 

The  cost  of  initial  proficiency  training  that  takes  place  in  Program  2 
units  is  somewhat  more  complex.  The  number  of  hours  the  unit  devotes  to  indi¬ 
vidual  training  on  a  particular  system  can  be  estimated  from  careful  study  of 
the  unit's  training  schedule.  A  pro  rata  share  of  the  unit's  annual  costs  of 
the  following  should  be  included: 

1 .  Spare  parts.  Rather  than  using  the  moment  of  equipment  failure 
or  repair  as  the  basis  for  attributing  total  repair  costs,  it 
would  be  more  accurate  to  charge  a  pro  rata  share  to  training 
based  on  the  fraction  of  the  use  time  allotted  to  training  or, 
if  known,  the  fraction  of  the  mileage  attributable  to  training, 
in  the  case  of  vehicles. 

2.  POL. 

3.  Anmunition  expended  for  training  on  the  system.  This  can 
probably  be  accounted  for  accurately  without  using  the  pro 
rata  distribution  (except  for  costing  by  task). 

4.  Military  pay  and  allowances  —  again  based  on  the  previous 
discussion  concerning  converting  to  dollars  the  time  diverted 
from  primary  mission. 

5.  Construction  of  special  ranges  or  training  facilities  (e.g., 

Redeye  Moving  Target  Simulator  (MTS))  for  training  on  the 
particular  system.  Here,  it  is  important  to  cost  only  the 
proportion  of  costs  attributable  to  initial  proficiency 
training. 

6.  Acquisition  costs  of  training  devices  charged  to  the  unit  and 
specifically  for  training  the  particular  system. 

7.  Operation  and  maintenance  of  training  devices  pertaining  to 
the  system  being  studied.  These  may  be  costs  incurred  by  the 
Training  Aids  Service  Center  (TASC). 

Costs  may  be  divided  further  into  variable  costs  and  fixed  costs.  Variable 
costs  as  used  herein  are,  in  general,  those  that  vary  with  changes  in  the  train¬ 
ing  load.  Thus  they  can  be  used  to  estimate  the  changes  in  total  cost  as  the 
training  load  changes.  Fixed  costs,  on  the  other  hand,  are,  within  certain 
limits,  insensitive  to  changes  in  the  training  load.  Support  by  Program  8T  TOE 
units  and  equipment  depreciation  are  examples  of  fixed  costs.  Ammunition  ex¬ 
penditures,  student  pay  and  allowances,  travel  pay,  and  per  diem  are  examples 
of  variable  costs. 

The  Litton  equations  constitute  the  cost  model  for  initial  training  of  a 
system.  The  equations  are  patterned  after  but  extend  those  developed  by  the 
Training  Analysis  and  Evaluation  Group  (TAEG)  of  the  U.  S.  Navy  and  reported  in 
TAEG  Report  Number  16,  "A  Technique  for  Choosing  Cost-Effective  Instructional 
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Delivery  Systems, "  April  1975.  The  Litton  model  employs  FORTRAN  representation 
for  the  names  of  the  cost  factors.  The  names  and  definitions  are  intentionally 
close  to  the  parallel  factors  employed  oy  TAEG.  The  basic  input  variable  selec¬ 
tion  philosophy,  also  similar  to  TAEG,  employs  five  generic  categories  of  cost 
factors: 

1 .  Facilities 

2 .  Equipment 

3.  Instructional  Materials 

4 .  Personnel 

5 .  Miscellaneous 

There  are  significant  differences  between  the  ‘two  models.  The  Litton  model 
expands  TECEP  coverage  of  institutional  training  costs  to  include  inter  alia 
institutional  field  training  costs.  The  Litton  model  also  adds  cost  factors  for 
the  individual  and  collective  initial  proficiency  training  done  in  Program  2 
(force)  units.  Thus,  a  number  of  TECEP' s  input  variables  have  been  emitted, 
and  new  variables  have  been  introduced  into  the  Litton  model. 

The  Litton  model  is  based  on  certain  assumptions.  Some  of  these  are  dis¬ 
cussed  in  the  explanation  of  the  cost  equations,  presented  in  the  User's  Guide 
that  accompanies  this  report  (Cost  and  Training  Effectiveness  Analysis  Perfor¬ 
mance  Guide) . 

The  Litton  model  assumes: 

1 .  Individual  training  beyond  initial  proficiency  on  the  system 
is  not  included.  Thus,  refresher  training  is  omitted. 

2.  Unit  effectiveness  training  beyond  initial  operational  capa¬ 
bility  is  not  included. 

3.  Costs  associated  with  loss  of  life,  property,  or  time 
occasioned  by  accidents  are  excluded. 

More  detailed  assumptions,  equations,  and  examples  of  use  of  the  model  are 
presented  in  the  CTEA  Performance  Guide  published  as  a  companion  to  this  report . 
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SECTION  V 

SYNTHESIS  OF  CTEA  METHODS 


The  LCShW  has  been  examined  to  determine  the  requirements  for  and  purposes 
of  CTEA.  This  examination  resulted  in  the  identification  of  probable  CTEA 
locations  in  the  LCSW1.  Methods  applicable  to  CTEA  have  been  identified 
through  a  literature  review  and  new  methods  have  been  devised  to  fill  gaps 
in  methodology. 

The  CTEA  methods  identified  through  the  literature  review  and  those  de¬ 
veloped  to  meet  recognized  needs  have  been  analyzed  to  reveal  their  embedded 
processes.  The  analysis  was  based  on  a  general  CTEA  model  that  identifies 
necessary  CTEA  processes  and  thus  points  to  a  set  of  input-data  situations  and 
to  a  taxonomy  of  CTEA  process  methods.  The  general  model  and  taxonomy  produce 
a  synthesis  of  methods  that  better  meet  the  needs  and  conditions  of  the  acqui¬ 
sition  process  than  any  single  method. 

A.  The  General  CTEA  Model 

The  selection  of  CTEA  methods  in  the  LCSMM  is  controlled  primarily  by  the 
available  data  and  the  issues  to  be  resolved  by  the  CTEA.  Method  selection 
begins  with  the  identification  of  the  basic  analytical  processes  made  necessary 
by  the  relative  sparseness  or  richness  of  the  data  and  then  proceeds  to  the 
selection  of  ways  to  accomplish  those  processes.  After  these  initial  selec¬ 
tions,  the  means  of  addressing  the  specific  issues  and  questions  of  the  CTEA 
are  considered. 

This  approach  to  the  selection  of  CTEA  methods  is  illustrated  in  Figure 
V-1 .  The  analyst  first  asks  whether  a  training  program  actually  exists.  The 
program,  including  tasks  to  be  trained,  methods  to  train  tasks,  personnel  to 
be  trained,  locations  of  training  for  various  tasks,  etc.  may  be  in  the  form 
of  a  training  description  not  yet  actualized  or  in  the  form  of  an  ongoing 
training  activity. 

Depending  on  the  answer  to  this  question,  the  analyses  proceeds  along  one 
of  two  primary  tracks.  If  there  is  no  training  program,  the  analyst  next  de¬ 
termines  whether  there  is  a  task  list  complete  enough  to  permit  predictions  or 
estimations  of  alternative  training  programs.  If  there  is  no  such  task  list, 
or  if  it  is  not  complete  enough  to  permit  the  prediction  or  estimation  of  al¬ 
ternative  training  programs,  he/she  selects  methods  for  the  accomplishment  of 
the  task-generation  process.  Once  a  task  list  has  been  generated,  or  it  has 
been  established  that  a  complete  enough  task  list  exists,  he/she  selects  methods 
for  the  training-prediction  process.  After  selection  of  such  methods,  methods 
are  selected  for  the  remaining  processes:  estimation  of  effectiveness,  costing 
of  training  program  alternatives,  cost  effectiveness  comparisons  of  training 
program  alternatives,  and  resolution  of  issues. 

If  there  is  a  training  program,  the  analysis  proceeds  along  the  other  pri¬ 
mary  track.  In  this  event,  the  second  question  concerns  the  existence  of  train¬ 
ing  program  alternatives.  If  there  are  no  alternatives  the  analyst  decides  how 
to  predict  or  estimate  alternatives  and  how  then  to  estimate  their  effectiveness. 
Once  it  has  been  established  that  alternatives  do  exist  or  methods  have  been 
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selected  for  the  prediction  of  alternatives  and  the  estimations  of  their  effec¬ 
tiveness,  the  analyst  next  determines  if  comparable  effectiveness  data  for  all 
alternatives  exist.  If  there  are  no  such  comparable  effectiveness  data  for  all 
alternatives  (i.e.,  comparable  performance  measurements  for  all  alternatives  or 
comparable  estimates  of  effectiveness  for  all  alternatives) ,  then  the  effective¬ 
ness  of  alternatives  for  which  there  are  no  data  or  performance  measures  are 
estimated.  This  requirement  is  based  on  the  assumption  that,  even  when  one  or 
more  alternatives  are  actual  training  activities,  the  analyst  will  not  be  able 
to  obtain  performance  measures  for  the  current  CTEA  by  requesting  the  admini¬ 
strations  of  tests  and  that  effectiveness  comparisons  of  alternatives  must 
therefore  include  comparions  of  estimated  effectiveness.  Once  it  has  been  de¬ 
termined  that  comparable  effectiveness  data  for  all  alternatives  exist,  or  methods 
for  estimating  the  effectiveness  of  all  alternatives  have  been  selected,  the 
analyst  selects  methods  for  the  remaining  processes:  cost  analysis  of  training 
program  alternatives,  cost/effectiveness  comparisons  of  training  program  alter¬ 
natives,  and  resolutions  of  issues. 

The  six  basic  input-data  situations  in  the  general  model  are: 

1.  No  Task  list  and  No  Training  Program.  In  the  worst  case,  function  and 
job  analyses  have  not  been  conducted  or  are  not  available  to  the  CTEA  analyst. 

Task  lists  and  training  programs  need  to  be  estimated  in  order  conduct  the  CTEA. 

2.  Task  List  but  No  Training  Program.  If  function  and  job  analyses  have 
been  completed  but  no  training  program  has  been  designed,  the  training  program 
needs  to  be  estimated  or  predicted  for  use  in  the  CTEA.  If  the  task  list  is 
very  rudimentary  (i.e.,  does  not  contain  complete  task  statements)  then  the 
analyst  needs  to  use  a  method  for  estimating  the  training  program  that  is  not 
overly  sensitive  to  gross  task  information.  A  training  program  can  be  estimated 
and  analyzed  from  a  rudimentary  task  list  but  the  result  will  be  a  rudimentary 
estimate.  Since  training  development  is  iterated  during  acquisition,  however, 

a  rudimentary  estimate  is  adequate  for  a  CTEA  early  in  the  LCSMM.  CTEA  methods 
used  with  early,  rudimentary  data  need  to  be  robust  in  the  face  of  vague,  in¬ 
complete,  and  inaccurate  task  information. 

If  this  situation  occurs  in  the  demonstration  and  validation  phase  or 
later,  precise  task  information  is  highly  desirable.  Some  task  and  training 
data  may  be  available  from  the  manufacturer,  who  develops  and  implements  train¬ 
ing  for  DT/OT  I.  Task  information  may  be  available  although  a  training  program 
may  not  be  available  in  printed  form.  The  CTEA  analyst  can  predict  a  training 
program  based  on  the  refined  task  information  and  the  predicted  program  is  a 
reasonable  estimate;  thus,  CTEA  estimates  based  on  the  predicted  training  are 
also  reasonable. 

3.  Training  Program  but  No  Alternatives  and  No  Effectveness  Data.  The 
training  program  implies  the  existence  of  a  detailed  task  list  that  may  contain 
complete  task  statements.  The  earliest  training  program  is  likely  to  be  the 
one  produced  by  the  manufacturer  although  it  may  be  a  service  school  product. 

Since  there  are  no  alternative  training  programs,  the  relative  merits  and  costs 
of  the  extant  program  have  not  been  estimated. 

1J.  Training  Program  with  Effectiveness  Data  but  No  Alternatives.  In  this 
instance  there  may  be  a  detailed,  complete  task  list.  The  training  program  is 
likely  to  be  that  produced  by  the  manufacturer,  and  the  effectiveness  data  are 
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likely  to  be  those  from  DT/OT  I.  These  data  are  likely  to  be  anecdotal  and  nay 
not  be  reported  since  training  evaluation  is  not  a  requisite  in  that  test.  The 
effectiveness  data  may  be  expert  estimates  or  judgments.  Since  there  are  no 
alternative  training  programs,  there  are  nc  comparative  estimates  of  cost  or 
effectiveness . 

5.  Alternate  Training  Programs  but  No  Effectiveness  Data  for  All  Alter¬ 
natives  .  The  programs  are  likely  to  be  those  produced  by  the  manufacturer  and 
by  the  service  school.  One  or  more  may  have  effectiveness  data  of  some  sort 
while  others  do  not.  For  example,  the  manufacturer’s  training  program  may  be 
tested  in  DT/OT  I  while  the  service  school  program  remains  untested.  Effective¬ 
ness  data  for  one  or  more  programs  need  to  be  produced,  perhaps  by  estimation, 
if  all  alternatives  are  to  be  compared  for  relative  effectiveness  and  cost. 

6. 

natives . 
training 

programs  may  have  been  tested  in  other  research.  The  effectiveness  measures 
may  differ  in  type  and  in  whether  they  are  empirical  or  judgmental.  This  sit¬ 
uation  is  likely  if  the  training  programs  were  tested  in  DT/OT  II  during  the 
full-scale  development  phase  of  the  LCSMM, 

The  general  CTEA  model  thus  identifies  the  total  set  of  required  CTEA 
processes  and  relates  these  processes  to  specific  CTEA  as  defined  by  their  in¬ 
put  data.  Because  each  process  may  be  carried  out  by  means  of  a  variety  of 
possible  methods,  each  process  suggests  a  category  of  a  taxonomic  scheme  through 
which  those  methods  may  be  analyzed  to  reveal  their  embedded  processes. 

B.  A  Taxonomy  of  CTEA  Processes 

All  of  the  CTEA-applicable  methods  reviewed  and  those  devised  by  Litton 
have  been  analyzed  to  determine  the  methods  through  whicn  the  required  CTEA 
processes  have  been  or  may  be  carried  out.  In  Table  V-1  the  required  general 
CTEA  processes  are  arranged  along  one  dimension  while  the  various  CTEA-applicable 
methods  are  arranged  along  the  other.  Each  column  constitutes  a  class  of  pro¬ 
cesses  in  which  each  method,  depending  on  data  conditions  and  other  factors,  may 
be  employed  to  carry  out  the  process  identified.  Rows  display  the  general  appli¬ 
cability  of  methods  to  CTEA. 

Table  V-1  shows  how  the  various  CTEA  and  CTEA-related  methods  fit  the  re¬ 
quirements  of  the  general  CTEA  model.  Unmet  needs  are  also  revealed.  The  one 
presently  available  automated  cost  model,  the  Navy’s  TECEP  cost  model,  does  not 
cover  the  costs  of  institutional  field  training  or  of  unit  training.  Available 
process  methods  for  U.e  prediction  of  training  programs  do  not  include  a  formal 
method  for  predicting  the  training  of  new  tasks  from  the  established  (and  thus  - 
validated)  training  of  similar  existing  tasks;  without  such  a  method  there  can 
be  no  formal  application  cf  the  accumulated  knowledge  of  how  tasks  of  various 
types  can  be  effectively  trained.  Finally,  there  are  no  methods  for  resolving 
the  issue  of  trainability.  How,  for  example,  does  the  analyst  determine  that  a 
task  may  nc be  trainable  or  may  pose  such  a  training  risk  that  operational  ef¬ 
fectiveness  may  be  jeopardized?  Litton ’s  trainability  analysis  is  offered  as 
an  interim  process  until  trainability  is  more  thoroughly  explored. 


Training  Program  Alternatives  and  Effectiveness  Data  for  All  Alter- 
This  situation  implies  a  fully  developed  task  list.  One  or  more 
programs  may  have  been  tested  to  yield  performance  measures,  and  the 
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TABLE  V-.1.  REQUIRED  ANALYTICAL  PROCESSES  AND  CANDIDATE  METHODS 


C.  CTEA  Situations  and  Strategies:  A  Synthesis  of  rrocesses 

The  general  CTEA  model  and  taxonomy  describe  a  variety  of  possible  CTEA 
situations  differing  in  input  information  and  issues  addressed.  he  situations 
correspond  roughly  to  the  phases  of  the  LCSM  but  do  not  correspond  exactly  be¬ 
cause  systems  do  not  progress  througn  the  acquisition  cycle  in  lock-step  manner. 
Potential  input  information  includes  task  lists,  training  programs,  and  effec¬ 
tiveness  data.  In  some  cases  the  data  may  exist  but  may  not  be  available  to 
the  CTEA  analyst  and  cannot  be  applied  in  the  analysis. 

The  final  challenge  in  the  synthesis  of  CTEA  methods  was  to  devise  general 
plans  or  strategies  to  guide  the  selection  of  processes  required  in  the  input- 
data  situations.  Each  situation  identifies  a  set  of  required  processes  and 
suggests  an  approach  to  meet  the  CTEA  objectives. 

1 .  Generation  of  Task  List 

CTEA  require  knowledge  of  what  is  to  be  trained,  but  those  conducted 
early  in  the  acquisition  process  may  not  have  access  to  such  knowledge,  at  least 
not  in  the  form  of  task  lists.  In  this  case  an  initial  effort  witnin  the  CTEA 
must  be  to  generate  task  lists.  Such  task  lists  need  not  be  as  detailed  as  the 
formal  task  descriptions  that  will  eventually  be  developed  (for  such  documents 
as  the  Soldier's  Manual),  but  they  should  be  detailed  and  comprehensive  enough 
to  permit  a  reasonably  reliable  estimation  (or  prediction)  of  the  training  pro¬ 
gram  that  will  support  the  tasks.  But  even  when  only  the  most  general  task 
information  can  be  obtained,  training  programs  can  be  at  least  tentatively 
estimated . 

The  person  responsible  for  an  early  CTEA  may  encounter  data  availability 
conditions  that  range  from  no  task  .list  at  all  to  existing  formal  task  lists. 

The  worst  case  occurs  when  there  is  general  information  about  the  new  system  but 
there  are  no  TASA  data.  If  a  formal  task  list  is  available  the  analyst  proceeds 
to  the  next  strategy  and  starts  with  training  generation. 

a.  The  Army  has  developed  these  methods  for  generating  task  lists  on 
the  basis  of  subject-matter  expertise: 

(1)  DIVAD  Gun  CTEA  Process  for  Generation  of  Task  List.  If  there 
are  no  TASA  data,  but  general  information  about  the  new  system  and  its  training 
devices  and  detailed  knowledge  about  similar  systems  is  available,  the  "generic 
CTEA"  processes  used  for  the  DIVAD  Gun  apply.  Training  program  alternatives  are 
predicted  on  the  basis  of  the  established  training  practices  of  existing  systems 
of  the  same  class  and  knowledge  of  the  new  system  and  its  training  devices.  The 
steps  are: 


o  Analyze  descriptions  of  new  system  and  place  new  system  in 
class  of  similar  systems. 

o  Identify  generic  tasks  of  class  of  systems.  (These  are 
tasks  or  task  types  that  tend  to  be  common  to  all  systems  of  the  class . ) 

o  On  the  basis  of  general  knowledge  of  new  systems  and  its 
training  devices,  revise  generic  task  list  to  make  it  as  specific  as  possible 
to  new  system. 
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A  task  list  generated  in  this  manner  is  used  to  estimate 
effectiveness  of  alternatives  rather  than  to  predict  them;  the  alternatives 
are  based  on  training  practices  of  the  class  of  weapon  systems. 

(2)  Hawk  Product  Improvement  Program  (PIP)  Training  Development. 
This  process  is  applicable  it  there  are  no  TA^A  data  but  tasks  are  inferred 
from  the  contractor’s  draft  technical  manual  and  from  detailed  knowledge  about 
the  antecedent  system.  It  was  applied  to  the  generation  of  task  lists  for  a 
PIP  and  may  be  limited  to  those  situations.  The  steps  are: 

o  Secure  task  lists  of  antecedent  system. 

o  Analyze  technical  documentation  (draft  TM)  and  compare 
with  task  lists  of  antecedent  system. 

o  Revise  (or  add  to)  task  lists  of  antecedent  system  to  make 
them  consistent  with  technical  documentation  of  new  (or  modified)  system. 

It  may  be  necessary  or  desirable  to  develop  some  new  tasks 
through  trial  performance  on  new  equipment. 

(3)  Roland  Training  Development.  The  Roland  training  develop¬ 
ment  process  is  applicable  if  there  are  TASA  data  and  descriptions  of  personnel 
functions  in  the  technical  documentation,  but  no  formal  lists  of  tasks.  It 
assumes  that  detailed  knowledge  of  the  antecedent  system  is  available.  This 
process  represents  a  considerably  higher  level  of  analysis  than  the  previous 
two  processes,  and  the  prediction  of  training  programs  may  therefore  be  more 
precise.  It  identifies  critical  tasks  and  selects  training  sites.  It  does, 
however,  require  an  analyst  who  is  a  subject-matter  expert  with  experience  and 
training  on  the  system  since  precise  judgments  are  required.  The  steps  are: 

o  Secure  contractor's  (manufacturer's)  LSAR  D  sheets  and 

draft  TM. 


o  Compare  tasks  in  LSAR  with  descriptions  in  TM  and  revise 
to  make  consistent  with  LSAR,  TM,  and  analyst's  expert  judgment. 

o  List  tasks  on  Task  Selection  Worksheet. 

b.  Strategy  for  CTEA  Lacking  Task  Lists.  Wien  there  is  no  task  list 
the  analyst  must  perform  aU  processes  from  generation  of  the  task  list  to 
resolution  of  issues.  In  this  situation  the  analyst  uses  expert  judgment  to 
generate  the  task  lists,  therefore  the  analyst  is  also  likely  to  use  expert 
judgment  for  prediction  of  alternative  training  programs  and  estimation  of 
training  effectiveness.  Formal  analytical  methods  also  can  be  used  to  predict 
the  training  programs  and  then  also  used  to  estimate  the  effectiveness.  The 
estimates  for  effectiveness  need  to  be  in  the  same  terms,  i.e.,  produced  by 
the  same  method,  to  be  comparable.  Similarly,  cost  estimates  for  comparisons 
need  to  be  in  the  same  terms. 

2.  Prediction  of  Training  Programs 

When  there  is  a  task  list  but  no  training  program  the  analyst  proceeds 
with  the  production  of  alternative  training  programs.  Prediction  of  training 
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programs  is  accomplished  in  several  ways.  First  is  expert  knowledge  of  the 
materiel  system  or  expert  knowledge  of  the  class  of  weapons.  Training  programs 
estimated  early  in  the  LCSM4  need  not  be  as  precise  as  those  estimated  later. 
Analytical  methods  such  as  TEEM,  TECEP,  and  the  Analogous  Task  Method  predict 
alternative  training  programs. 

a.  These  methods  are  recommended  in  particular  if  there  is  one  train¬ 
ing  program  already  designed  but  there  are  no  alternative  programs  because 
these  methods  are  recomn ended  for  the  comparison  of  alternative  training 
programs. 


(1)  TEEM  Process  for  Predicting  Training  Programs.  TEEM  predicts 
training  programs  by'  describing  both  tasks'  and  certain  elements  of  training  pro¬ 
grams  in  terms  of  the  same  set  of  variables ,  called  a  "metalanguage."  The  vari¬ 
ables,  or  metalanguage,  were  identified  through  a  literature  search  and  generally 
represent  research  findings  concerning  relationships  between  task  condition  or 
requirements  and  various  types  of  instructional  media  and  methods.  Variables 
relating  task  conditions  or  requirements  to  content  and  content  structure  have 
also  been  tentatively  identified,  but  these  are  not  yet  employed  in  the  processes 
of  the  method. 


Tasks  are  first  described  in  terms  of  variables  classified  as 
stimulus  variables,  response  variables,  and  feedback  variables.  They  are  then 
logically  grouped  so  that  media  selection  may  proceed.  Each  task  results  in  the 
selection  of  one  or  more  media  devices  or  forms  ideally  suited  to  that  task,  but 
the  total  set  of  such  devices  or  forms  is  reduced  in  response  to  real-world  con¬ 
straints  so  that  only  a  few  media  devices  or  forms  serve  all  tasks.  The  func¬ 
tional  contexts  of  task  groups  are  similarily  described  in  terms  of  method  vari¬ 
ables  ( these  relate  task  contexts  to  instructional  methods) ,  and  instructional 
methods  are  thus  selected. 

The  input  of  this  training-program-prediction  process  is  task 
information  in  various '  formats  from  which  at  least  nouns,  verbs,  and  modifiers 
descriptive  of  the  task  may  be  derived.  Briefly,  the  task  information  must  in¬ 
dicate  what  is  done  to  what  within  what  limits  or  under  what  conditions.  The 
output  is  a  set  of  media  and  instructional  methods.  Further  development  of  the 
method  may  result  in  the  inclusion  of  specifications  for  content  and  content 
structure. 


(2)  TECEP  Process  for  Predicting  Training  Programs.  TECEP  pre¬ 
dicts  training  programs  in  terras  of  suitable  instructional  algorithms  and  de¬ 
livery  systems.  Each  task  or  training  objective  of  a  training  program  is  matched 
with  one  of  12  learning  algorithms.  The  matching  is  accomplished  by  comparing 
the  statement  of  the  task  or  training  objective,  including  the  verbs,  with  the 
lists  of  action  verbs,  behavioral  attributes,  and  examples  of  training  objec¬ 
tives  included  in  the  algorithm  charts.  Once  each  task  has  been  matched  with 

an  algorithm,  all  tasks  are  grouped  or  classified  according  to  the  algorithm 
selected  for  it.  Then,  two  or  more  types  of  delivery  systems  are  selected  for 
each  group  of  tasks .  Such  delivery  systems  are  to  be  considered  only  candidates 
for  the  developing  training  program,  however,  and  must  survive  tests  of  practi¬ 
cality  and  other  constraints. 

(3)  piVAD  Gun  Process  for  Predicting  Training  Programs.  This 
process  method  might  be  termed  the  "worst-case  method1*  for  the  prediction  of 
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training  program  alternatives.  Training  program  alternatives  are  formulated 
on  the  basis  of  established  training  practices  within  the  class  of  weapon 
systems  (or  other  materiel  systems)  to  which  the  developing  system  belongs. 
Alternatives  are  likely  to  oonsist  of  different  arrangements  of  operational 
equipment  and  various  training  devices  (simulators,  etc.). 

(*♦)  Analogous  Task  Method  for  Predicting  Training  Programs.  Tasks 
of  fielded  systems  that  are  analogous  to  the  tasks  for  which  training  programs 
are  being  predicted  (target  tasks)  are  identified.  Once  it  has  been  verified 
that  an  analogous  task  is  similar  enough  to  the  target  task  to  permit  the  in¬ 
ference  that  the  same  training  program  would  be  effective  in  training  both 
tasks,  the  training  program  of  the  analogous  task  becomes,  in  effect,  the 
training  program  of  the  target  task. 

b.  Strategies  for  CTEA  Using  Predicted  Training  Programs.  Training 
programs  may  be  estimated  by  informal  judgment  or  by  f'ormal  analytical  methods 
such  as  TEEM,  TECEP,  and  ATM.  The  formal  analytical  methods  are  recommended  to 
predict  training  program  alternatives  to  be  compared  with  either  those  predicted 
by  expert  judgment  or  with  existing  training  programs.  Large  differences  in  the 
training  programs  predicted  by  divergent  methods  signal  areas  that  require  closer 
attention. 


If  at  least  one  training  program  has  been  predicted,  informal  ex¬ 
pert  knowledge  is  not  recomnended  for  prediction  of  alternatives.  Since  the 
tasks  are  known,  the  formal  analytic  methods  can  be  employed.  Whatever  the 
method  employed,  different  levels  of  training  devices  and  media  are  estimated 
for  comparison  with  the  devices  and  media  in  the  existing  program. 

3.  Estimation  of  Training  Effectiveness 

Ideally,  effectiveness  is  determined  empirically,  but  because  of  the 
impossibility  of  obtaining  direct  empirical  data  for  systems  not  yet  developed 
to  the  prototype  stage  and  the  difficulties  in  obtaining  it  even  for  systems 
in  prototype  or  later  stages,  estimation  is  often  necessary. 

a.  Three  kinds  of  methods  for  the  estimation  of  effectiveness  appear 
to  be  possible:  analysis  of  how  well  a  proposed  training  program  "fits"  the 
tasks  to  be  trained;  expert  judgments  about  the  relative  effectiveness  of  a 
set  of  proposed  programs;  and  generalization  of  the  demonstrated  effectiveness 
of  the  training  program  of  one  task  to  the  training  program  of  all  nearly  iden¬ 
tical  (analogous)  tasks. 

(1)  TEEM  Process  for  Estimating  Effectiveness.  In  this  method 
an  efficiency  metric  that  expresses  how  well  a  training  program  "fits"  the  tasks 
to  be  trained  is  taken  as  an  estimate  of  effectiveness.  The  metric  is  the  ratio 
of  measures,  derived  from  the  media  selection  processes,  of  constrained,  real- 
world  programs  and  unconstrained,  ideal  programs.  That  is,  each  task  results 
in  the  initial  selection  of  a  media  type  ideal  for  that  task,  but  then  tasks 
are  logically  grouped  and  the  total  set  of  media  is  reduced  to  an  average,  com¬ 
promised  media  selection  for  each  functional  task  group  or  training  situation, 
simply  because  real-world  constraints  would  not  permit  the  costly  and  possibly 
unmanageable  ideal.  Thus,  the  measures  or  scores  associated  with  the  selec¬ 
tion  of  the  constrained,  real-world  set  of  media  are  divided  by  the  measures 
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or  scores  associated  with  unconstrained,  ideal  set  of  media  to  produce  an 
"efficiency”  ratio.  The  assumption  is  that  if  a  training  program  were  totally 
unconstrained  in  the  selection  of  media,  then  the  media  type  selected  for  each 
task  would  be  ideal  for  that  task  and  the  effectiveness  of  the  training  would 
be  greater  than  if  the  program  were  constrained  and  media  selections  were  less 
than  ideally  suited  to  each  task. 

(2)  DIVAD  Gun  CTEA  Process  for  Estimating  Effectiveness.  This 
process  method  for  the  estimation  0/  effectiveness  cannot  be  described  in  de¬ 
tail  because  of  a  lack  of  information,  but  it  was  clearly  informal  as  applied 
in  the  DIVAD  Gun  CTEA.  One  or  more  analysts,  experts  on  the  class  of  weapons 
to  which  the  DIVAD  gun  belongs,  examined  the  proposed  alternatives  in  light  of 
their  knowledge  of  the  "generic”  tasks  of  the  class  of  weapons  and  much  knowl¬ 
edge  as  they  had  of  the  DIVAD  Gun  and  the  ranked  them  on  the  basis  of  judged 
effectiveness . 

(3)  Analogous  Task  Method  for  Estimating  Effectiveness .  Tasks  of 
fielded  systems  that  are  analogous  to  the  tasks  for  wnich  training  programs  are 
being  predicted  (target  tasks)  are  first  identified.  The  training  prgrams  of 
the  analogous  tasks  are  adopted  as  the  training  programs  of  the  target  tasks, 
and  the  empirical  effectiveness  data  of  the  analogous  tasks  become  estimates  of 
the  effectiveness  of  the  analogous  task  training  program  applied  to  the  target 
tasks. 


(4)  TRAINVICE  Method  for  Estimating  Effectiveness.  To  estimate 
the  effectiveness  or  proposed  training  device  alternatives,  the  CTEA  analyst 
applies  the  TRAINVICE  procedures  for  the  following  analyses: 

0  Task  commonality 

0  Physical  similarity 

o  Functional  similarity 

o  Learning  deficit 

o  Training  techniques 

For  each  subtask  to  be  trained  by  a  device  these  yield  an 
estimate  of  device  effectiveness,  and  the  separate  estimates  are  then  combined 
to  yield  an  overall  estimate  of  device  effectiveness.  (According  to  Training 
Device  Requirements  Documents  Guide,  DARCOM  provides  cost  estimates  for  train¬ 
ing  devices  though  the  device  project  office  of  the  proponent  school.  DTD  may 
be  required  tc  provide  some  cost  data.  It  is  also  important  to  note  that, 
since  there  is  not  yet  a  way  to  combine  the  effectiveness  estimation  metrics 
of  both  training  devices  and  training  programs  to  yield  a  single  metric  as  an 
estimate  of  the  effectiveness  of  a  training  program  with  a  training  device, 
the  estimation  of  the  effectiveness  of  training  devices  and  the  estimation  cf 
the  effectiveness  of  training  programs  must  be  considered  as  is-ues. 

Once  the  most  effective  training  device  alternatives  have  been  selected,  how¬ 
ever,  it  may  be  considered  within  the  context  of  training  program  alternatives 
as  the  costs  and  effectiveness  of  these  alternatives  are  estimated.  Given  the 
characteristics  of  the  most  effective  training  device  alternatives,  TCA,  for 
example,  ccuid  be  used  to  estimate  the  relative  effectiveness  cf  training  pro¬ 
grams  witr.  ar.d  witr.cut  the  devic  . ' 
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(5)  TCA  Process  for  Estimating  Effectiveness.  In  TCA  the  estimate 
of  effectiveness  is  the  Training  Consonar  Ratio  (TCfc).  Tasks  are  described 
in  terms  of  65  psychological  variables  a  aen  each  combination  of  medium  and 
methoa  is  described  in  terms  of  the  same  d5  variables.  The  goodness-of-fit 
between  the  task  descriptions  and  the  descriptions  of  the  combinations  of  met;*; 
and  methods  used  to  train  the  tasks  is  expressed  in  terms  of  the  Training  Con-, 
sonance  Ratio.  The  TCP.  is  almost  exactly  analogous  to  the  efficiency  ratio  of 
TEEM  since  the  same  lists  of  variables  are  used  in  essentially  the  same  way. 

b.  Strategies  for  Estimation  of  Training  Effectiveness.  If  -nere  are 
alternative  training  programs  but  not  all  of  the  programs  have  effectiveness 
data,  the  analyst  starts  with  effectiveness  estimation. 

It  is  reasonable  to  apply  expert  judgment  to  estimate  effectiveness 
of  training  programs  that  were  predicted  by  expert  judgment.  If  a  task  is  simi¬ 
lar  to  a  task  in  an  existing  system  about  which  the  analyst  has  knowledge,  and 
if  it  is  known  that  a  certain  type  of  training  results  in  the  required  level  of 
proficiency  on  that  task,  then  that  training  approach  can  be  estimated  for  the 
new  training  program. 

If  the  analogous  task  method,  a  formal  analytic  procedure,  was  used 
to  predict  a  training  program,  then  it  is  also  used  to  estimate  the  training 
effectiveness  of  the  program.  If  TEEM  was  used  to  estimate  two  or  more  training 
alternatives,  the  TEEM  efficiency  ratio  is  the  estimate  of  effectiveness.  Train¬ 
ing  consonance  analysis  (TCA)  can  be  used  to  estimate  effectiveness  no  matter 
how  the  programs  were  predicted:  the  training  consonance  ratios  produced  by 
TCA  are  the  effectiveness  estimates.  TCA  estimates  effectiveness  of  both  ex¬ 
isting  and  predicted  programs.  It  also  diagnoses  training  deficiencies,  ex¬ 
cesses,  and  redundancies  for  revision. 

The  state  of  the  task  information  (e.g.,  very  rudimentary,  refined, 
or  fully  developed)  determines  the  precision  of  training  program  estimation  and 
training  effectiveness  estimation.  Within  a  comparison,  the  levels  of  informa¬ 
tion  concerning  training  effectiveness  must  be  common  for  them  to  be  comparable. 

4 .  Cost  Analysis  of  Training  Programs 

The  estimation  of  the  costs  of  training  programs  is  a  complex  process 
involving  many  variables,  but  estimation  must  be  precise  as  possible  since  cost 
effective  decisions  about  competing  alternatives  are  to  be  made.  The  one  cost 
process  method  selected  for  inclusion  in  the  manual  appears  to  offer  the  possi- 
bilitiy  of  accurate  cost  analysis  along  with  a  reasonable  approach  to  the  com¬ 
plexity  of  the  costing  problem. 

a.  Litton  Cost  Model.  The  Litton  cost  model  is  a  modification  of  the 
TECEP  cost  model.  It  expands  the  coverage  of  training  costs  to  include  both 
institutional  field  training  and  initial  proficiency  training  in  the  units,  and 
it  should  therefore  provide  more  precise  analysis  of  training  costs  than  TECEP 
cost  model.  Because  it  is  computerized  the  complexity  of  the  problem  from  the 
analyst's  point  of  view  is  somewhat  lessened.  The  Litton  cost  model  is  intended 
for  cost  estimates  only,  not  for  budgetary  requirements. 
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b.  Strategy  for  Cost  Analysis  of  Training  Programs.  The  modified 
TECEP  model  is  recommended  for  all  CTEA  situations  in  the  LCSMM.  If  one  train¬ 
ing  program  is  estimated  by  expert  judgtient,  or  if  one  program  actually  exists, 
while  the  others  are  estimated  by  using  the  formal  analytic  methods,  it  is  im- 
por' ant  to  note  that  the  cost  estimates  must  be  comparable  in  level  of  detail 
and  precision  if  valid  cost  comparisons  among  alternatives  are  to  be  made. 

5.  Comparison  of  Alternative  Training  Programs 

Once  the  effec . -veness  and  costs  of  competing  training  program  alter¬ 
natives  have  been  estimated  or  measured,  the  alternatives  must  be  compared  with 
each  other  and  with  externally  imposed  standards  or  criteria.  The  purpose  of 
the  comparison  is  the  selection  of  the  most  cost  effective  of  the  alternatives 
which  meet  the  criteria.  In  order  to  assure  that  the  conflicting  demands  of 
adequate  effectiveness  and  minimal  costs  are  met,  some  sort  of  rational  scheme 
c  method  for  comparison  must  be  employed. 

In  established  practice,  schemes  or  methods  for  comparison  range  from 
informal  comparisons  of  rankings  of  alternatives  on  both  effectiveness  and 
costs  to  formal  cost  benefit  analyses.  Further,  because  it  may  be  necessary 
or  desirable  to  consider  factors  in  addition  to  cost  and  effectiveness  in  the 
selection  of  a  training  program,  there  are  methods  for  incorporating  these  ad¬ 
ditional  factors  in  the  decision  metric. 

a.  The  following  process  methods  by  no  means  represent  all  possi¬ 
bilities  for  the  comparison  of  alternatives,  but  they  do  represent  established 
methods  and  provide  for  the  requirements  of  most  CTEA. 

(1)  TEEM  Comparison  of  Training  Alternatives .  In  this  method  the 
estimate  of  effectiveness ,  the  efficiency  ratio,  is  not  a  strong  one  but  it  is 
the  best  available  ir.  a  very  early  CTEA.  Cost  effectiveness  (C/E)  ratios  are 
f.rmed  and  compared.  The  analyst  selects  the  efficiency  ratio  that  represents 
the  lowest  acceptable  effectiveness  and  this  step  is  difficult  and  judgmental. 
The  plot  of  the  series  of  efficiency  ratios  (Figure  III-7)  is  closely  examined 
for  pronounced  decreases  in  efficiency  as  a  result  of  the  reduction  of  the  media. 
Efficiency  may  drop  off  slowly  at  first  as  medium  after  medium  is  removed  from 
the  training  description  but  then  at  some  point  it  begins  to  drop  off  quite 
sharply.  At  this  point  or  sooner  the  analyst  will  probably  decide  that  minimal 
acceptable  efficiency  has  been  reached  and  that  the  media  set  can  be  reduced  no 
further.  Then,  because  the  relationship  between  cost  and  efficiency  (and  there¬ 
fore  effectiveness)  is  not  likely  to  be  linear,  he/she  will  choose  media  sets 
represented  by  several  points  higher  than  the  minimal  point  and  then  compare  the 
media  sets  in  terms  of  cost  and  efficiency.  The  alternative  with  the  smallest 
acceptable  c/E  value  is  chosen  since  it  represents  an  acceptable  efficiency 
(effectiveness)  at  the  lowest  cost. 

(2)  TCA  Comparison  of  Training  Alternatives.  The  estimate  of 
effectiveness  in  Training  Consonance  Analysis  (TCA)  is  the  Training  Consonance 
Ratio  (TCR).  The  TCR  is  similar  to  the  efficiency  ratio  of  TEEM,  but  it  is 
derived  for  training  programs  predicted  by  means  other  than  formal  analytic 
methods.  TCRs  are  derived  by  comparing  task  descriptions  with  training  program 
descriptions.  The  analyst  encodes  task  descriptions  and  training  program  de¬ 
scriptions  as  input  for  the  TCA  computer  program,  and  the  computer  program 
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provides  output  in  the  form  of  TCRs  and  other  data  including  diagnostic  infor¬ 
mation.  Training  program  alternatives  (i.e.,  media  sets  and  methods)  are  com¬ 
pared  as  TCRs  combined  with  costs  (C/E  ratios). 

When  a  predicted  training  program  is  being  compared  with  an 
actual  (implemented)  training  program  and  the  actual  program  has  not  demon¬ 
strated  acceptable  effectiveness,  the  predicted  training  programs  may  have  both 
a  higher  TCR  and  a  lower  C/E  value.  In  this  case,  several  approaches  are  open 
to  the  analyst.  First,  the  predicted  program  could  be  recornnended  for  replace¬ 
ment  of  the  actual  program.  Second,  the  description  of  the  predicted  program 
could  be  compared  task  by  task  with  the  description  of  the  actual  program  to 
form  a  basis  for  recornnended  revisions  of  the  actual  training  program.  Third, 
the  diagnostic  information  provided  by  the  TCA  program  (training  deficiencies, 
excesses,  and  redundancies)  could  also  be  used  to  recommend  revisions  of  the 
actual  training  program.  The  second  and  third  approaches  would  involve  alter¬ 
natives  of  training  descriptions,  recomputations  of  TChs  and  costs,  and  further 
comparisons  of  C/E  values. 

(3)  BDM/CARAF  Comparison  of  Training  Alternatives.  The  essence 
of  this  method  is  the  comparison  of  training  program  alternatives  in  terms  of 
multiple  measures  of  training  effectiveness  (MOTE).  Each  MOTE  selected  is 
weighted  according  to  its  perceived  importance,  and  a  total  weighted  effective¬ 
ness  score  (the  standard  measure  of  effectiveness)  is  derived.  Costs  are  then 
derived  or  estimated  and  C/E  values  compared .  The  method  is  suitable  only  for 
actual  training  programs  when  interval  or  ratio  data  are  available. 

(4)  DIVAD  Gun  Comparison  of  Training  Alternatives.  This  method 
is  appropriate  when  the  choice  in  terras  of  cost  and  effectiveness  is  very  clear 
cut.  It  is  actually  a  very  simple  version  of  the  DRIMS  CTEA  Method,  but  when 
the  choice  is  obviously  very  clear  cut,  it  is  completely  adequate  and  avoids 
unncossary  complexity. 


(5)  DRIMS  CTEA  Comparison  of  Training  Alternatives.  This  method 
of  comparison  considers  factors  in  addition  to  cost  and  effectiveness  that  may 
be  considered  important  to  the  selection  of  a  training  program  or  training  pro¬ 
gram  element.  While  it  appears  to  be  most  relevant  to  the  selection  of  a  train¬ 
ing  device  (as  it  was  applied  in  the  DRIMS  CTEA),  it  could  be  applicable  to  whole 
training  programs  as  well.  Alternatives  are  ranked  on  all  selection  factors, 
including  one  or  more  in  addition  to  cost  and  effectiveness,  and  a  total  rank¬ 
ing  score  is  derived.  Selection  is  based  on  the  total  ranking  score. 

(6)  Analogous  Task  Method  Comparison  of  Training  Alternatives. 

When  more  than  one  analogous  task  is  discovered,  the  effectiveness  data  asso¬ 
ciated  with  the  training  program  of  one  analogous  task  is  compared  with  the 
effectiveness  data  of  the  training  program  of  the  others.  The  analogous  task 
training  program  that  is  most  effective  becomes,  by  inference,  the  training 
program  of  the  target  task. 


(7)  TECEP  Comparison  of  Training  Alternatives.  TECEP  processes 
lead  directly  to  training  alternative  comparisons.  Given  the  training  objec¬ 
tives,  TECEP  selects  learning  algorithms  and  instructional  delivery  systems. 

The  next  step  of  the  TECEP  Process  (Figure  III-5)  is  estimation  of  the  training 
system  cost,  and  the  final  step  is  selection  of  a  cost-effective  training  de¬ 
livery  system.  , 
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b.  Strategies  for  Comparison  of  Alternative  Training  Programs.  If 
some  training  programs  have  effectiveness  data  while  others  do  not 7  the  ones 
that  have  effectiveness  data  are  considered  separately.  Training  programs  that 
have  performance  measures  for  each  task  are  examined  to  select  those  tasks  ac- 
ceotable  in  cost  and  effectiveness.  The  decision  is  based  on  the  cost  effec¬ 
tiveness  ratios  showing  the  programs  with  the  lowest  values  (lowest  cost  per 
unit  of  effectiveness). 

Cost  effectiveness  ratios  are  the  estimated  cost  divided  by  the 
estimated  effectiveness.  These  ratios  are  used  to  compare  the  training  pro¬ 
grams.  Analytic  methods  such  as  TEEM  indicate  which  levels  are  acceptable  and 
which  are  eliminated.  Final  selection  of  a  training  program  is  deferred  until 
all  issues  have  been  resolved.  For  example,  a  training  program  with  a  higher 
cost  effectiveness  ratio  may  be  desirable  because  it  is  more  effective  in 
training  high-risk  tasks. 

If  the  effectiveness  data  are  not  performance  measures  or  if  they 
are  not  based  on  standards  in  task  and  training  objectives,  the  analyst  uses 
Training  Consonance  Ratios  (TCR).  TCR  provide  estimates  for  comparison  of  the 
training  programs  on  the  basis  of  effectiveness  and  costs.  Tentatively  the 
program  with  the  lowest  cost/effectiveness  ratio  is  selected.  The  relation¬ 
ships  among  the  cost/effectiveness  ratios  are  informative.  If  the  TCR  of  one 
or  more  predicted  alternatives  is  higher  than  the  ratio  for  an  existing  pro¬ 
gram,  the  analyst  compares  the  media  and  methods  of  the  predicted  alternatives 
task  by  task  with  the  media  and  methods  of  the  existing  program.  Also,  the 
analyst  examines  the  existing  and  predicted  training  program  diagnostic  infor¬ 
mation.  This  information  indicates  revisions  of  the  existing  program  that  will 
increase  its  TCR  and  thus  its  estimated  effectiveness.  The  values  may  warrant 
adoption  of  one  of  the  predicted  alternative  programs. 

If  some  training  programs  exist  and  the  others  are  estimated,  and 
one  or  more  of  the  programs  has  measures  based  on  task  and  training  objectives, 
the  analyst  uses  the  cost/effectiveness  ratios  differently.  The  training  pro¬ 
gram  or  programs  having  effectiveness  measures  are  compared  to  the  objectives 
to  assess  deficiencies.  If  standards  or  criteria  indicate  a  deficiency,  the 
existing  program  is  eliminated  from  selection  in  its  unrevised  state.  The  TCR 
is  computed  for  the  program  revised  as  recommended  on  the  basis  of  the  diag¬ 
nostic  data,  and  the  ratio  is  evaluated.  The  lowest  cost/effectiveness  ratio 
is  selected,  with  the  caveat  that  the  cost  estimates  for  existing  and  predicted 
programs  may  not  be  comparable.  Costs  of  predicted  alternatives  must  include 
all  development  costs  while  the  costs  of  existing  programs  include  fewer  de¬ 
velopmental  costs  or  none  at  all. 

If  the  election  is  made  of  an  existing  program  with  excesses  or 
redundancies,  the  TCA  diagnostic  information  is  used  to  recommend  revisions 
and  thus  increase  the  cost/effectiveness. 

If  only  some  of  the  alternative  training  programs  have  effective¬ 
ness  data,  and  the  data  are  performance  measures,  the  TCRs  are  used  to  compare 
all  alternatives.  If  a  training  program  without  performance  measures  has  a 
higher  TCR  than  a  program  with  performance  measures,  that  program  may  be  more 
effective  than  the  program  with  measures.  The  analyst  might  reconmend  collec¬ 
tion  of  empirical  data  to  verify  the  effectiveness.  If  the  measures  Of  per¬ 
formance  include  more  than  one  measure  of  training  effectiveness,  the  analyst 
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can  use  BBM/CARAF  to  compute  a  total  weighted  effectiveness  score  and  use 
that  score  to  form  a  cost  effectiveness  ratio. 

When  there  are  training  program  alternatives  and  data  for  all 
alternatives,  the  first  step  is  to  eliminate  from  consideration  any  alterna¬ 
tives  that  have  unacceptable  effectiveness .  TCRs  can  be  used  to  diagnose  prob¬ 
lems  and  recommend  revisions.  The  acceptable  programs  are  then  compared  in 
cost/effectiveness.  Programs  with  the  lowest  values  are  selected.  If  factors 
other  than  performance  measures  are  to  be  considered,  the  analyst  uses  the  DRIMS 
CTEA  method. 


Though  the  purpose  of  comparison  is  to  select  the  most  cost  effec¬ 
tive  program,  a  CTEA  may  oonclude  with  the  reconmendation  that  final  selection 
be  deferred  until  certain  issues  are  resolved  through  further  investigations. 

Such  investigations  typically  involve  experimental  determinations  of  the  effects 
on  proficiency  of  training  program  alternatives. 

fc>.  Resolution  of  Issues 

Most  CTEA  issues  are  resolved  through  processes  from  the  generation  of 
task  lists  to  the  comparison  of  alternative  training  programs.  Still,  some  will 
not  be  resolved  by  these  processes,  and  special  methods  for  their  resolution 
must  be  selected  or  developed. 

a.  The  typical  issues  requiring  special  methods  are: 

(1)  Trainability.  The  issue  of  trainability  requires  that  tasks 
and  training  programs  be  compared  with  personnel  characteristics  to  determine 
if  the  personnel  who  are  to  man  a  system  can  be  satisfactorily  trained  to  man 

it  through  proposed  or  actual  training  programs.  According  to  a  TRADOC  official 
(Telephone  Interview,  25  June  I960)  precise  methodology  in  this  area  is  now 
practically  nonexistent  but  under  development.  This  official  endorsed  the  train- 
ability  analysis  proposed  by  Litton  (see  Section  IVB)  as  a  satisfactory  stopgap 
until  more  precise  methods  are  developed. 

(2)  Training  Device  Requirements.  Several  of  the  methods  tested 
above  address  training  devices  as  elements  of  training  programs,  but  none  con¬ 
siders  simulators  or  similar  training  devices  on  the  basis  of  the  several  kinds 
of  fidelity.  Because  of  the  large  number  of  uncertainties  in  the  specifications 
of  training  devices  and  the  incomplete  development  of  methods  for  resolutions 
of  training  device  questions,  this  issue  is  considerd  to  be  suitable  work  for 

a  behavioral  scientist  rather  than  the  typical  CTEA  analyst,  and  no  training 
device  anaxynis  method  has  been  selected  or  recommended. 

(3)  Performance  Versus  Standards.  This  issue  will  be  resolved 
through  the  first  five  processes  if  complete  task  statements  are  available  or 
generated,  but  the  development  of  standards  is  a  lengthy  and  difficult  process 
usually  requiring  threat  analysis  and  therefore  does  not  appear  to  be  a  suit¬ 
able  undertaking  for  the  typical  CTEA  analyst.  No  methods  beyond  those  included 
under  Process  A  have  been  selected  or  included  in  the  manual. 

(4)  Performance  Versus  Hardware.  This  issue  is  an  aspect  of  the 
trainability  issue.  If  one  or  more  tasks  are  found  or  estimated  to  be  untrain- 
able,  the  problem  may  be  attributed  to  the  hardware  (faults  in  design,  extremely 


difficult  tasks  for  personnel  regardless  of  characteristics,  etc.),  but  no  par¬ 
ticular  method  for  the  resolution  of  this  issue  has  been  selected  or  Included 
in  the  manual. 

(5)  Performance  Versus  Personnel  Selection.  This  issue,  too.  is 
an  aspect  of  tt  trainability  issue.  It  may  be  that  one  or  more  tasks  found  or 
estimated  to  \  untrainable  given  one  set  of  personnel  characteristics  may  in¬ 
deed  be  trains. .e  given  another  set  of  personnel  characteristics.  Because  of 
the  undeveloped  state  of  methodology  in  this  area,  no  particular  method  for 
considering  personnel  selection  within  a  CTEA  has  been  selected  or  included  in 
the  manual. 


(b)  Other  Possible  Issues.  These  include  such  issues  as  user 
acceptance  of  a  training  device,  least-cost  location  of  training  program,  and 
the  extent  of  or  impact  of  skill  decay. 

b.  Methods  that  assist  in  the  resolution  of  the  issues  are: 

(1)  ITV  CTEA.  The  ITV  CTEA  resolved  the  issue  of  training  loca¬ 
tion  (e.g.,  which  Army  training  center)  through  cost  comparisons  of  various 
locations.  Then,  under  an  equal-effectiveness-of-alteraatives  assumption, 
alternatives  are  compared  and  a  selection  made  on  cost  alone. 

(2)  PRIMS  CTEA.  The  DRIMS  CTEA  resolved  the  issue  of  user  accep¬ 
tance  by  ranking  each  device  and  including  the  ranking  in  the  overall  effective¬ 
ness  assessment. 

(3)  Trainability  Analysis.  Task  descriptions  and  training  pro¬ 
grams  are  compared  with  available  data  on  personnel  characteristic.  On  the 
basis  of  this  informal  (subjective)  comparison,  the  analyst  rates  the  risk  that 
a  task  cannot  be  adequately  trained  as  (1)  high,  (2)  moderate,  or  (3)  low. 

Then  in  the  same  way  tasks  are  compared  with  mission  analyses  or  operational 
concepts  and  the  criticality  of  each  task  is  judged  to  be  (1)  high,  (2)  moderate, 
or  (3)  low.  Where  high  risk-high  criticality;  tasks  coincide  great  concern  for 
trainability  is  indicated;  where  moderate  risk-moderate  criticality  tasks  coin¬ 
cide  moderate  concern  is  indicated;  and  where  low  risk-low  criticality,  high 
risk-low  criticality,  or  low  risk-high  criticality  tasks  coincide  little  or  no 
concern  is  indicated  (Figure  IV -4) . 
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SECTION  VI 


SUMARY 


—/’Powerful  methods  applicable  to  CTEA  and  the  whole  manpower,  personnel,  and 
training  side  of  acquisition  via  the  LCSM  have  been  developed  and  await  broad 
application  and  refinement.  A  number  of  these  methods  have  -been .described,  in 
Section  III.  . y 

C  Existing  methods  were  weak  in  three  areas.  First,  lack  of  adequate  defini¬ 
tion  of  analogous  tasks  degraded  use  of  historical  data  from  fielded  weapon  sys¬ 
tems.  Second,  the  issue  of  trainability  (a  critical  question  in  the  LCSM)  was 
not  adequately  addressed.  Third,  TECEP,  the  most  thorough  cost  model  applicable 
in  the  LCSM4,  did  not  address  training  in  Army  units.  Litton' s  methods,  devised 
to  assist  Army  analysts  in  these  three  areas,. hare  beerupresented.in  Section  IV. 

_ " _ _ 

Section  V  describes  a  general^ model  for  CTEA  In  "the  LCSMiC— The-mcdel  * 
queries  the  data  available  regardless  of  the  point  in  the  acquisition  cycle. 

It  is,  designed  to  answer  the  questions  that  are  pertinent  at  each  stage  in  the 
cycle.  The  model  guided  selection  of  methods  for  use  by  Array  analysts  and 
strategies  for  application  of  the  methods,  ^ The  details  concerning  strategies 
and  procedures  are  presented  in  a  companion] volume,  the  CTEA  Performance  Guide. 


"  Methods  that  include  the  consolidated  data  base  are  especially  interesting 
since  they  provide  a  formal  informational  point  of  contact  for  the  many  agencies 
involved  in  an  acquisition.  When  Army  acquisitions  routinely  make  use  of  a  eon- 
,  solidated  data  base,  both  the  timeliness  and  precision  of  the  analyses  will  be 
greatly  enhanced.  As  the  various  agencies  and  personnel  involved  in  an  acquisi¬ 
tion  contributed  data  to  and  retrieved  it  from  a  consolidated  data  base,  they 
>  would  necessarily  become  much  more  aware  of  the  relatedness  of  the  various  ac¬ 
quisition  processes.  While  data  not  yet  produced  would  still  not  be  available, 
j  such  data  as  had  been  produced  would  be  available  to  all  who  need  it.  Another 
|  possible  benefit  of  the  consolidated  data  base  is  that  it  could  encourage  all 
i  involved  in  the  acqusition  process  to  speak  the  same  technical  language,  thus 
\  increasing  the  efficiency  and  precision  of  communications. 


"'^Powerful  CTEA  and  acquisition  process  methods  of  the  future,  if  they  are 
realized  by  the  Array,  will  probably  all  be  computer  based.  This  feature  promises 
not  only  greatly  enhanced  efficiency  in  the  processing  of  information  but  almost 
certainly  a  degree  of  precision  not  now  achievable.  It  is  possible  that  such  an 
increase  in  the  efficiency  and  precision  of  information  processing  could  both 
increase  the  effectiveness  of  systems  and  reduce  the  time  required  to  acquire 
them. 
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APPENDIX  B 


ABSTRACTS  OF  SELECTED  COST  OF  TRAINING  STUDIES 


Several  documents  pertaining  to  cost  analysis  of  training  were  reviewed, 
The  documents  most  relevant  to  CTEA  in  the  LCSM-1  were  abstracted  and  the  ab¬ 
stracts  are  presented  in  this  Appendix. 


THE  ARMY  LIFE  CYCLE  COST  MATRIX 


(Department  of  the  Army  Pamphlets:  No.  11-2,  "Research  and  development  cost 
guide  for  Army  materiel  Systems,”  May  1976;  "Investment  Cost  Guide  for  Army 
Materiel  Systems,"  April  1976;  No.  11-4,  "Operating  and  Support  Cost  Guide  for 
Army  Materiel  Systems,"  April  1976;  No.  11-5,  "Standards  for  Presentation  and 
Documentation  of  Life  Cycle  Costs  Estimates  for  Army  Materiel  Systems,"  May 
1976.  Washington,  DC:  Hq,  Department  of  the  Army) 


In  order  to  encompass  all  the  costs  incurred  in  the  life  cycle  of  a  weapon 
system  (or  major  component  of  the  system) ,  the  Department  of  the  Army  has  issued 
a  number  of  cost  guides  (DA  Pams.  11-2,  11-3,  11-4,  and  11-5)  for  life  cycle 
costing  of  materiel  systems.  The  cost  elements  are  as  shown  in  Figure  B— 1 ,  Army 
Life  Cycle  Cost  Matrix.  Although  a  cost  attributable  to  the  requirements  for 
training  may  be  costed  in  another  category,  this  cost  matrix  treats  training 
costs  as  follows: 


Category 

Budget 

1.0 

Research  and  Development 

1.0b 

Training 

RDTE,  OMA 

2.0 

Investment 

2.0b 

Training 

PROC.,  OMA 

3.0 

Operating  and  Support  Cost 

3.023 

Unit  Training,  Ammo  &  Missiles 

PROC 

3.061 

Personnel  Replacements 

MPA,  OMA  (depending 
on  focus,  assumptions, 
etc. ) 

These  are  further  defined  by  equations  (Figs.,  B-2,  B-3,  3-4,  B-5,  B-6). 
Supporting  these  equations  (a  gross,  aggregate  level)  are  detailed  documenta¬ 
tions  of  each  cost  element,  justifying  the  dollar  figure  proposed  (for  format 
see  Fig.  B-7).  Although  all  life  cycle  costs  (LCC)  are  covered  by  this  matrix, 
problems  remain.  A  graphic  display  of  the  issues  is  shown  in  Fig.  B-6.  Another 
issue,  not  shown  in  the  table,  is  the  inability  of  the  analyst  specializing  in 
one  area  to  isolate  and  aggregate  all  the  cost  relevant  to  his  particular  focus. 
Training  is  an  outstanding  example  of  such  an  area.  For  the  purpose  of  fore¬ 
casting  for  complete  LCC  of  training  for  a  given  system  the  analyst  must  have 
some  cost  data,  such  as  ammunition  and  POL  costs,  that  the  weapon  system  LCC 
includes  as  separate  categories.  The  training  system  cost  analyst  usually  takes 
the- 'bottoms  up’  costing  approach  shown  in  Fig.  B-8.  This  is  likely  the  case 
when  conducting  a  CTEA  for  a  developing  system  even  though  the  cost  information 
will  be  used  as  input  to  an  analysis  conducted  by  a  system  cost  analyst  as  re¬ 
quired  at  all  major  decision  points  in  the  acquisition  process  (usually  a  formal 
COEA) .  This  problem  was  recognized  by  Hawley  and  Thomason  (1978),  developer's  of 
an  Air  Defense  CTEA  methodology  especially  fitted  within  the  Army’s  LCSMM  and 
TSM  concepts.  Making  recommendations  for  future  research,  they  recognize  re¬ 
quirements  for: 
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Army  Life  Cycle  Cost  Matrix 


smm  iip»  at  mmmmm 


Figure  B-3.  Investment  Training  Costs 


Figure  B-4.  Cost  Per  Man  Year  Methodology 


CONSUMPTION.  3.02 

UNIT  TRAINING,  AMMUNITION  AND  MtSSILES.  3.023 


Sourct:  DA  Pirn  11—4,  p.  6-3 


Figure  B-5.  Operating  Phase,  Unit 
ammunition  Costs 


INDIRECT  SUPPORT  OPERATIONS,  3.06 
PERSONNEL.  REPLACEMENT,  3.061 


PERSONNEL 

REPLACEMENT 

3.061 


TOTAL  QUANTITY 
OPERATIONAL 
EQUIPMENT 


TOTAL  NUMBER 
MILITARY  PERS 
PER  EQUIPMENT 


AVERAGE 

ANNUAL 

ATTRITION 

RATE 


AVERAGE 

REPLACEMENT 

COST/MAN 


NUMBER 

OPERATING 

YEARS 


Sourct :  DA  Pam  1 1  -4,  p,  6-7 

Figure  B-6.  Operating  Phase,  Personnel  Replacement  Costs 
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TOP-DOWN  COSTING  APPROACH 


1.  More  definitive  work  on  cost  analysis  and  resource  impact  projection. 

A  major  criticism  of  the  costing  model  (e.g.,  TECEP)  used  in  the  current  demon¬ 
stration  project  is  that  the  input  cost  categories  do  not  match  the  output  cate¬ 
gories  of  the  Army's  accounting  system.  Future  work  should  involve  a  resolution 
of  this  problem.  The  concurrent  and  predictive  validity  of  costing  procedures 
also  has  not  been  established. 

2.  More  work  should  be  done  on  establishing  the  organizational  links 
necessary  to  obtain  information  concerning  RAM,  maintenance  cost  data  and  main¬ 
tenance  policy  information  for  complex  training  devices.  Equipment  RAM  and 
maintenance  considerations  often  underlie  assumptions  concerning  the  operating 
characteristics  of  the  materiel  system  and  the  training  program.  Therefore, 
the  correct  estimation  of  such  parameters  is  essential  for  valid  cost  estima¬ 
tion  and  later  sensitivity  analyses. 

3.  The  investigation  of  cost  analysis  methods  providing  expected  values 
and  confidence  limits  as  output.  In  CTEA,  it  would  be  useful  to  obtain  ndt 
only  a  point  estimate  of  cost,  but  also  a  statement  of  the  precision  or  relia¬ 
bility  of  the  cost  point  estimate. 


TRADOC  TEA  HANDBOOK 


(Dept,  of  Array ;  "TRADOC  Training  Effectiveness  Analysis  Handbook,  First  Draft" 
White  SandM  Missile  Range,  NM:  US  Army  TRADOC  Systems  Analysis  Aotivity,  un¬ 
dated.  Received  in  1979) 


TRADOC  includes  CTEA  as  one  type  of  a  family  of  training  effectiveness 
analysis  (TEA) .  In  the  reoent  TRADOC  Training  Effectiveness  Analysis  Handbook. 
First  Draft  (1979)  a  CTEA  is  said  to  "evaluate  the  cost  and  effectiveness  of 
alternative  training  approaches  as  they  are  being  formulated  to  support  the 
developing  total  hardware  oriented  system"  (p  1-7).  The  study  further  speci¬ 
fically  defines  a  CTEA  as  "a  study  that  assesses  the  variable  effectiveness 
and  variable  costs  associated  with  a  set  of  pre-defined  alternative  training 
subsystems  at  each  major  deoision  point  in  the  acquisition  process."  (p  2-2) . 

This  definition  forms  the  basis  of  the  relation  of  costing  to  the  CTEA 
methodology.  As  shown  in  Fig.  B-9,  costing  is  specifically  addressed  at  two 
points .  The  first  occasion  is  near  the  end  of  the  preliminary  phase  after  the 
concept  design  of  alternative  training  systems.  Estimated  concurrently  with 
training  subsystems'  effectiveness,  the  cost  of  each  alternative  system  is  input 
to  the  decision  process  that  chooses  the  best  alternatives.  The  second  costing 
effort  takes  place  between  OT  I  and  OT  II.  At  this  time,  a  smaller  set  of  'best 
alternatives'  are  selected  and  alternative  training  subsystems  designed. 

This  set  is  costed  and  their  effectiveness  assessed  and  a  still  more 
select  set  chosen  and  tested  in  OT  II.  Costing  effort,  however,  is  implied  — 
either  new  or  revised  estimates  —  throughout  the  system  and/or  training  sub¬ 
system  life.  The  TRADOC  Hanabook  suggests  using  the  var  able  cost,  variable 
effectiveness  model  as  the  most  appropriate  for  comparing  alternative  subsystems. 
They  state  that  the  second  cost  analysis  will  be  more  detailed  since  both  train¬ 
ing  and  hardware  subsystems  ere  approaching  final  form  making  more  precise  esti¬ 
mates  possible. 

The  cost  analysis  process  is  treated  in  a  separate  chapter,  Chapter  10, 

Cost  Analysis. 

The  TRADOC-suggested  cost  methodology  Js  shown  in  Fig.  B-10.  It  is  applic¬ 
able  to  both  costing  ' points ' .  Cost  consideration  and  data  collection  are  as¬ 
sumed  to  begin  at  the  front-end  analysis. 

The  generalized  cost  model  is 

TTC  a  ITC  +  UTC 
or 

Total  training  cost  s  institutional  training  cost  plus  unit  training  cost 

The  cost  analyst  is  directed  to  work  with  the  rest  of  the  study  team  to 
define  the  essential  elements  of  analysis  and  to  identify  cost  subelements  that 
they  require  (e.g.,  ammo,  courses,  MOS  to  be  trained).  Once  the  TTC  of  each 
alternative  is  calculated,  the  cost  analyst  is  instructed  to  highlight  in  his 
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analytical  resentation  aubelements  of  high  impact  on  total  coat  and  those  whose 
incremental  or  decremental  change  will  influence  system  effectiveness.  Critical 
cost  considerations  may  outweigh  total  costs  of  the  alternatives  and  may  be  the 
deciding  facers  when  TTC  of  the  alternatives  do  not  differ  significantly.  Of 
further  importance,  this  oost  methodology  implies  that  within  the  baaio  model, 
subordinate  equations  will  be  written  baaed  on  the  institution* 1  training  con¬ 
cepts,  the  unit  training  oonoepts,  the  training  system  requirements  specifica¬ 
tion  and  the  life  cyole  oost  estimates.  LGC  elements  (LCCE)  are  those  diaoussed 
earlier.  TRADOC  identifies  the  LCCE  as  the  primary  building  block  in  the  oost 
analysis  and  the  'only  tool’  the  analyst  has  to  define  the  hardware  oosts  of  the 
training  systems.  Suggested  sources  (Army  Agunciea)  are  also  listed  in  Annex  I. 

Although  the  list  of  data  for  institutional  training  does  not  explicitly 
state  that  field  as  well  aa  base  (school)  training  are  included,  field  training 
and  its  costs  are  not  precluded  and  they  oould  be  included  in  such  categories 
as  equipment  used,  etc. 

Unit  training  is  examined  by  'training  event'  (e.g.,  ARTEP  event?'  and  the 
cost  of  each  event  over  the  life  of  the  system  accumulated.  This  should  amount 
to  a  close  approximation  of  Litton’ s  model  for  LCC  cost.  No  provision  is  ex¬ 
plicitly  stated  for  prorating  costs  or  evaluating  remaining  assets  at  end  of 
costing  period. 

The  TRADOC  system  rests  on  identifying  critical  tasks.  The  task  list, 
procured  from  the  study  proponent  and/or  contractor  or  devised  by  the  CTEA 
analysts,  is  established  in  the  front-end  analysis  (Step  1.0,  Fig.  B-9).  This 
list  is  used  to  design  the  first  set  of  alternative  training  subsystems  but 
explicit  directions  for  developing  a  task  list  are  not  specified. 


SYSTEMS  APPROACH  TO  TRAINING  (SAT) /P-1  (1975) 


(Ring,  W.  A  Half ,  H.  &‘1ayatyna  approach  to  training,  final  report.  Appendix 
A;  Coat  details.  SAT- \ ,  Vol.  2,  krlght-Patterson  AFB7  OH:  B-1  Systems  project 
Office,  Data  Configuration  Diviaion,  July  1975) 


In  1975  Ring  et  al.  applied  the  teohniquea  of  ayatems  analyais  to  Instruc- 
tional  System  Development  (ISD)  to  ensure  that  the  entire  training  system  is 
included  in  Training  System  Design  (TSD).  The  SAT  was  used  to  oevelop  a  recoat- 
mended  alternative  training  system  for  the  future  B-1  bomber.  A  computerized 
aid  to  the  ISD  was  usea  —  the  Train!;.*  Resources  Analytic  Model  (TRAM).  TRAM 
provides  time-phased  LC  costs  and  resource  requirements  for  the  training  system 
of  the  B-1.  The  SAT  approach  provides  for  trade-offs  and  sensitivity  analyses 
of  such  features  as  ratio  of  hours  spent  using  various  training  media,  centrali¬ 
zation  of  training  facilities,  ratio  of  airborne  to  ground-based  training  hours, 
and  training  media  time  phasing.  Outputs  of  the  program  are  descriptions  of  the 
reconmended  and  alternative  training  systems  with  a  syllabus  for  each  course, 
descriptions  of  required  media  and  facilities,  costs,  and  schedules. 

The  initial  source  of  information  is  considered  to  be  the  task  analysis 
data  base.  The  authors  state  that  the  quality  of  this  data  base  establishes 
the  potential  quality  of  the  overall  instructional  system. 

The  SAT  process  was  applied  to  the  aircrew  only.  It  is  not  known  if  main¬ 
tenance,  ground  crew  or  other  training,  also  part  of  the  overall  B-1  LCC  were 
similarly  analyzed.  The  task  analysis  identified  1500  task  elements. 

The  recommended  system  school  setting  augmented  with  training  devices  and 
the  alternative  training  systems  were  all  costed.  These  costed  systems  were 
identified  by  both  TRAM  and  a  quick- look  version,  TROLIE  (Training  Resources 
Organized  for  Logical  Integration  of  Expenses)  model  runs  and  subsequent  data 
development . 

The  cost  moael  was  developed  to  fit  a  particular  segment  of  the  B-1  re¬ 
lated  personnel  —  the  four  aircrew  stations,  and  three  particular  training 
alternatives  —  three  types  of  special  trainers  for  each  crew  station.  The 
data  identified  as  required  to  cost  each  of  these  special  trainers  by  crew 
station  were: 

Initial  Acquisition  Costs 

RDT&E  and  other  preproduction  costs  (includes  cost  of  first  unit). 

These  costs  were  all  estimated  through  application  of  factors  that 
were  percentages  of  the  hardware  acquisition  cost.  For  example, 
initial  training  s  1.5#  of  hardware  acquisition  cost  (excluding 
familiarization  trainer) . 

First  unit  production  costs  for  trainers.  These  costs  were  first 
determined  to  lie  within  ranges  and  then  estimated  as  a  point  within 
the  range.  For  example,  the  procedures  for  pilot/copilot  station 
are  estimated  to  fall  infia  0.10  -  0.25  ($  x  10°)  range  and  esti¬ 
mated  to  be  0.20  ($  x  10°). 
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General  purpose  carrels  and  briefing  rooms.  These  costs  were  deter¬ 
mined  to  be  a  function  of  required  multimedia  A/V  equipment  and  cost 
of  classroom  new  construction  or  conversion.  Cost  factors  such  as 
$40/sq.  ft.  for  new  construction  were  used. 

Facility  requirements  and  costs.  A  similar  costing  approach  to  that 
of  the  carrels  and  briefing  rooms  was  used. 

Instructional  material.  Representative  costs  for  instructional 
material  preparation  and  production  such  as  viewgraphs  at  $10  each; 
programmed  courses,  self-paced,  at  $600  per  instruction  hour;  and 
technical  data  manuals  at  $125  per  page  were  used. 

Annual  Operations  and  Maintenance  Costs 

Instructor  personnel 

Trainee  TDY 


B-1  Aircraft  (direct  costs) 


Equipment  operations  and  maintenance 
Equipment  Upgrade 
Instructional  material 


Facilities 


All  of  these  costs  were  estimated  using  factors  for  costs  and  average  lead 
times. 

A  further  feature  of  this  costing  effort  was  the  assignment  of  a  data  qua¬ 
lity  designator  to  each  cost  element.  Costs  used  were  related  on  a  scale  of  1-3 
depending  on  the  quality  (predicted  accuracy)  of  the  data  source,  (See  Table  B-1 
for  a  list  of  all  elements  and  quality  estimates.)  Among  the  cost  data  sources 
were  manufacturers;  government  facilities  and  simulation  users;  the  Simulation 
Technology  Assessment  Report  (STAR);  the  Simulator  Special  Project  Office;  com¬ 
mercial  handbooks  and  annuals  of  A/V  and  other  media  and  construction  costs;  and 
USAF  Cost  and  Planning  Factors  (APR  173-10). 

A  baseline  system,  three  modifications  of  a  part  mission  trainer  (a  simu¬ 
lator)  alternate  baseline  concepts,  variations  in  crew  ratio  and  variations  in 
the  replacement  model  were  costed. 

The  foregoing  methodology  and  basic  data  were  used  to  derive  costs  used 
to  evaluate,  comparatively,  the  three  training  systems.  Output  was  formulated 
to  display: 

Non-Recurring  Costs 

RD&E 

Acquisition  Facilities 

Facilities 

Instructional  Material 
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Recurring  Support 
Operation  and  Maintenance 


Coats  were  displayed  as  totals  by  year 
for  the  first  year  and  inflated  dollars  for 
for  ten  years  of  operation  were  costed. 


as  appropriate  in 
succeeding  years. 


current  dollars 
Life  cycle  costs 


Table  1*1.  Data  Quality  Estimates 
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3 
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3 
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3 
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3 
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3 
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2 
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2 
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2 
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3 
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3 
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1 
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1 
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2 
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3 
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2 
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1 
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2 
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2 
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3 
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3 
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INSTRUCTOR  PERSONNEL 

1 
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1 
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2 
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3 

EQUIPMENT  UPGRADE 
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INSTRUCTIONAL  MATERIAL 
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2 
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2 

A  TECHNIQUE  FOR  CHOOSING  COST-EFFECTIVE  INSTRUCTIONAL 
DELIVERY  SYSTEMS,  TAEG  REPORT  NO.  16  (1976) 


(Braby,  R.,  Henry,  J.M.,  Parris,  W.F.,  Jr.  &  Swope,  W.M.  A  Technique  for  Choosing 
Cost-Effective  Instructional  Delivery  Systems,  TAEG  Rep.  No.  16).  Orlando,  FL. 
Dept,  of  the  fclavy,  Training  Analysis  and  Evaluation  Group,  April  1975) 


One  of  a  series  of  reports  prepared  for  the  Navy's  Training  Analysis  and 
Evaluation  Group  in  Orlando,  FL,  TAEG  Report  No.  16  presents  a  technique  for 
choosing  cost-effective  delivery  systems  for  proposed  training  programs.  It  is 
not  designed  as  a  CTEA  methodology  although  it  could  be  used  for  conducting  one. 
The  researchers  (Braby  et  al.,  1975)  call  the  technique  TECEP  —  training  effec¬ 
tiveness,  cost  effectiveness  prediction.  It  is  intended  for  use  by  the  training 
system  designer  during  the  concept  design  phase.  The  methodology  includes  re¬ 
finements  of  two  earlier  efforts  ~  one  with  the  same  objective  (TAEG  Report  No. 

1,  1972)  and  a  technique  for  choosing  cost  effective  instructional  media  (1974). 
TAEG  Report  No.  16  is  designed  to  be  used  with  two  other  reports  —  TAEG  Report 
No.  23,  Learning  Guidelines  and  Algorithms  for  Twelve  Types  of  Training  Objectives 
and  TAEG  Report  No.  24,  Choosing  Instructional  Delivery  Systems  with  the  TECEP 
Technique. 

The  report's  system  development  model  is  earlier  shown  in  Section  III,  Figure 
III-5.  It  requires  user  expertise  (unlike  the  BDM  model  designed  for  CTEA).  The 
position  of  the  TECEP  in  the  TSDM  (a  continuous,  reflexive  model)  is  as  shown.  A 
diagram  of  the  TECEP  (an  input-process-output)  model  is  shown  in  Figure  III-4. 

As  in  other  methodologies  discussed,  the  costs  are  estimated  after  the 
instructional  systems  have  been  designated.  The  authors  stress  the  importance 
of  defining  study  objectives.  For  example,  if  the  objective  is  to  select  the 
most  cost  effective  alternative  from  among  several  choices,  the  resources  common 
to  all  in  similar  amounts  can  be  factored  out.  When  absolute,  life-cycle  cost 
is  required,  then  all  resources  must  be  included  and  evaluated  at  their  oppor¬ 
tunity  cost. 

The  basic  output  of  the  TECEP 's  cost  module  is  the  current  cost  of  each 
alternative  proposed  training  system.  Additional  output  was  (at  the  time  of 
the  study  --  may  now  be  increased)  formatted  to  show  the  total  and  average 
yearly  cost  per  student  position,  average  cost  per  course  graduate,  and  a 
distribution  of  incidence  of  cost  over  the  alternative's  life. 

The  designers  recognize  that  it  will  be  necessary  to  access  multiple  data 
sources  to  apply  the  cost  model.  They  suggest: 

1 .  past  records  of  operational  units 

2.  NAVPERS  personnel  data 

Model  limitations  are  seen  to  be:  The  model  cannot  select  the  most  effi¬ 
cient  media;  cannot  forecast  total  system  cost;  assumes  all  variable  cost  func¬ 
tions  are  linear;  does  not  provide  for  evaluating  secondary  effects. 
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The  inputs  are  divided  into  seven  categories:  facilii 
structional  material,  personnel,  supplies,  students,  and  m. 
in  all).  This  model  is  computerized,  written  in  FORTRAN  r 
or  selected  items  may  be  chosen.  A  maximum  20-year  life  m 


TRAINING  DEVELOPERS  DECISION  AID  (TDDA)  FOR  OPTIMIZING 
PERFORMANCE- BASED  TRAINING  IN  MACHINE  ASCENDANT  MQS  (1978) 


(Pieper,  W.J.,  Guard,  N.R.,  Michael,  W.T.  4  Kordek,  R.S.  Training  Developers 
Decision  Aid  for  Optimizing  Performance-Baaed  Training  in  Machine  Aaoendant  MQS. 
Valencia,  PA:  Applied  Science  Associated,  IncT,  T57ST 


The  TDDA  is  a  process  for  generating  training  specifications  for  developers 
who  must  decide  what,  where  and  hew  tasks  within  an  MOS  must  be  trained.  It  was 
developed  in  two  phases  —  a  manual  process  model  based  on  logic  charts  and  a 
computerized  model  for  which  the  developer  inputs  task  data.  Three  outputs  are 
available  —  training  prescriptions,  training  hierarchies  and  sequences,  and 
training  costs.  A  single  assumption,  the  availability  of  a  usable  task  list  is 
made. 


Cost  analysis  is  the  last  module.  The  authors'  (Pieper  et  al.,  1978) 
rationale  is  that  this  facilitates  identification  of  beneficial  cost  trade¬ 
offs  and  permits  selection  of  only  the  necessary  and  relevant  cost  statistics. 
Training  material,  course  POI,  criterion-referenced  tests  and  OJT  plans  are 
all  considered. 

This  cost  model  differs  from  those  previously  discussed.  Training  methods 
are  assigned  values  derived  through  a  cost  rating  technique.  Pieper  et  al. 
chose  this  method  because:  (1)  actual  dollar  costs  vary  over  time  while  rela¬ 
tive  difference  in  costs  remain  essentially  constant;  (2)  ratings  can  include 
items  such  as  student  time  without  conversion  to  dollar  amounts;  and,  (3)  the 
various  methods  of  instruction  can  be  ranked  on  relevant  cost  dimensions. 

The  tasks  of  any  one  MOS  were  treated  as  a  single  set  —  some  taught  in 
schools,  some  on-the-job,  and  a  few  given  no  formal  training.  In  OJT  two  fac¬ 
tors,  number  of  hours  in  the  learning  cycle  and  number  of  hours  to  learn  a 
task,  are  their  determinants.  The  two-year  promotion  cycle  is  set  as  the  learn¬ 
ing  cycle,  with  each  year  treated  uniquely  and  time  for  training  calculated 
using  factors  for  such  items  as  non-productive  duty  and  movement  found  in  the 
Army  Regulations.  This  process  was  recommended  for  OJT  calculation  for  each 
MOS. 


Training  in  schools  was  treated  differently.  Here  Pieper  et  al.  looked  at 
the  direct  and  indirect  costs  of  establishing  and  operating  a  resident  training 
facility.  They  categorized  training  into  seven  method  cost  classes: 

Class  1.  Conventional  classroom,  demonstration,  case  study,  and 
guided  discussion. 

Class  2.  Peer  tutor. 

Class  3 .  Tutoring . 

Class  4.  Programmed  instruction  (student  and  program  paced), 
games,  in-basket,  and  study  assignment  book. 
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Class  5.  Traditional  practical  exercises. 

Class  6.  Programmed  practical  exercise. 

Class  7.  Computer  assisted  instruction. 

Next,  they  placed  the  direct  and  indirect  cost  variables  into  seven  classes. 
These  were  expected  to  vary  as  a  result  of  training  method  changes.  The  cost 
variables  used  were: 

1 .  Square  footage 

2.  Instructor  to  student  ratio 

3.  System  equipment 

4 .  Furnishings 

5 .  Expendable  supplies 

6.  Training  aid  development 

7.  Training  material  development 

Third,  the  method  cost  classes  were  ranked  on  each  of  the  cost  variables. 

An  illustration,  Table  B-2,  shows  the  method  cost  classes  ranked  on  one  cost  vari¬ 
able,  value  or  square  footage  required  for  that  instructional  method.  (This  is 


value  or  benefit  not  dollar  cost.) 

Method  Cost  Class 

Sq  Ft 

1 

1 

2 

3.5 

3 

3.5 

4 

3.5 

5 

6.5 

6 

6.5 

7 

3.5 

Table  B-2 


As  explained  by  the  authors,  "the  cost  variables  were  assigned  ranks  on 
the  basis  of  their  estimated  relative  values  on  a  scale  of  one  to  seven.  In 
the  example  (Table  B-2) ,  the  square  footage  variable  for  traditional  classroom 
training  was  assigned  a  value  of  one,  because  this  method  can  accommodate  more 
students  per  square  foot  that  any  of  the  other  methods.  The  square  footage 
variable  for  peer  tutor,  tutor,  programed  instruction,  and  computer  assisted 
instruction  training  method  classes  were  each  assigned  a  value  of  3.5,  because 
there  was  no  detectable  difference  in  the  amount  of  room  needed  for  their  use. 
These  four  tied  rankings  consumed  2,  3,  4,  and  5  on  the  one  to  seven  scale. 

The  square  footage  variable  for  traditional  practical  exercise,  and  programmed 
practical  exercise  training  methods  were  assigned  a  value  of  6.5,  because  there 
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was  no  detectable  difference  in  the  amount  of  room  needed  to  use  them.  These 
two  tied  rankings  consumed  positions  six  and  seven  on  the  scale,  and  completed 
the  rankings  for  the  square  foot  variable." 

All  the  cost  variables  were  assigned  rank  values  using  this  procedure.  The. 
seven  method  cost  classes  and  seven  cost  variables  were  then  arranged  in  a  table 
(Table  B-3) . 

Using  these  seven  rankings  a  mean  rank  for  each  method  cost  class  can  be 
computed.  The  remaining  calculations  were: 

-'After  the  mean  ranks  were  computed,  the  conditions  for  determining 
the  relation  of  the  cost  intervals  to  the  rank  intervals  was  set. 

The  lowest  cost  method  was  assigned  a  value  of  one,  and  the  highest 
cost  method  a  value  of  two.  Since  the  number  of  intervals  within  a 
range  of  whole  numbers  is  equal  to  the  highest  number  in  the  range 
minus  the  lowest  number  in  the  range,  the  cost  interval  for  Table  B-2 
is  2-1  si ,  and  the  rank  interval  of  the  seven  variable  scale  is  7-1=6. 

The  relationship  of  the  one  cost  interval  to  the  six  rank  intervals 
is  the  ratio  of  1/6  which  is  .167.  This  ratio  was  used  as  a  constant 
to  compute  the  cos*  multiplier  for  the  mean  ranks  of  the  method  cost 
classes . 

The  mean  rank  cost  multiplier  *  y •  in  Table  B-2  was  derived  by  sub¬ 
tracting  one  from  the  mean  rank,  multiplying  the  difference  by  the 
cost  ratio  and  adding  one,  (e.g.,  y  =  (rank  -  1  (x))  +  1).  The 
number  of  hours  of  each  method  within  the  training  cost  option  was 
multiplied  by  its  cost  factor  to  get  the  Method  Cost  Indicator  (MCI) 
for  the  methods.  The  MCt  for  all  methods  within  the  training  option 
were  then  summed  yielding  a  Resident  Option  Cost  Indicator  (RMCI  = 

ROCI) ." 

With  the  exception  '■*  AI,  the  cost  method  classes  and  cost  variables  were 
also  applied  to  OJT  and  ai.  ualogous  on-the-job  option  cost  indicator,  the  JOCI 
was  computed. 

The  two  were  then  summed  to  obtain  a  cost  indicator  for  each  training  op¬ 
tion  (TOCI) . 


TOCI  =  ROC*  JOCI 
or 

Training  option  cost  indicator  =  Resident  cost 
option  indicator  +  job  option  cost  indicator 


The  TOCI  is  used  to  establish  a  ratio  of  one  training  cost  option  to  another 
giving  the  training  manager  as  indication  of  relative  cost.  This  enables  the 
manager  to  select  the  most  promising  option  or  options  for  a  complete  CTEA  effort 
The  procedures  have  been  developed  in  both  computer  and  manual  mode. 
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Tabl*  B-3.  Cost  Ranking 


MITHOO  COST  CUM 

B 

IMT/ 

STU 

SYS 

IQ 

B 

Q 

l  1 

MAT 

iff 

a 

1.  TRAOITIONAL 
CLASSROOM 

1.0 

n 

1.1 

H 

B 

H 

3.0 

3.4 

141 

2.  MIR  TUTOR 

EB 

Kl 

KB 

3.0 

KB 

KB 

1.S 

3.1 

1.33 

3.  TUTOR 

EB 

mm 

KOI 

3.0 

KB 

KB 

1.S 

KB 

1.40 

4.  PROGRAMMED 
INSTRUCTION 

E9 

El 

BW 

n 

S.0 

MB 

1.43 

S.  TRADITIONAL 
PRACTICAL 

IXKRCISK 

H 

n 

B 

1.0 

H 

B 

B 

B 

1.83 

6.  PROGRAMMED 
PRACTICAL 
IXERCISE 

H 

a 

■ 

fl 

H 

B 

B 

B 

1.72 

7.  COMPUTER 
ASSISTEO 
INSTRUCTION 

■ 

H 

7.0 

B 

B 

B 

B 

1.74 

Source  Pioptr  tt  «(.,  p.  11-48 
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PREDICTION  OF  TRAINING  PROGRAMS  FOR  USE  IN  CTEA  (1978) 


(Jorgensen,  C.C.  4  Hoffer,  P.L.  Prediction  of  Training  Program  for  Use  In  Cost 
Training  Effect iveneas  Analysis.  Alexandria,  VA:  U.sT  Army  Research  Institute 
for  the  Behavioral  and  Social  Seienoea ,  1978) 


This  report  describes  a  modal  davalopad  by  Jorgensen  and  Hof far  (1978)  that 
uses  a  systems  approach  to  deflna  training  alamants  and  thair  intaractiona  within 
a  CTEA  satting.  Following  a  sat  of  logio  flowcharts,  tha  stapa  in  CTEA  conduct 
are  dapictad.  Initial  input  is  task  and  analysis  data  (idaally  but  not  neoes- 
sarily  in  ISD  form).  Tha  modal  haa  baan  coaputaritad.  Tha  modal  output  ia  a 
cost-effectiveness  ratio,  obtainad  by  dividing  tha  astiaatad  dollar  oost  of  a 
program-generated  training  program  by  tha  program*  a  estimated  affaotlvanaaa  ("an 
afficianoy  metric  is  Msad  to  estimate  training  af fectivenaaa .  Tha  efficiency 
metric  is  an  indication  of  tha  overall  bast  fit  between  tha  training  require¬ 
ments  of  the  task  and  the  ability  of  various  madia  and  methods  to  satisfy  these 
demands.") 

The  system  overview  was  shown  earlier  in  Figure  III-1 .  As  mentioned  pre¬ 
viously,  the  methodology  begins  with  a  task  list  or  description  that  states  the 
tasks  and  subtasks.  After  the  tasks  have  been  described,  grouped  and  appropriate 
training  methods  and  media  selected  (all  by  analyst/ computer  interaction)  the 
selected  alternative  training  programs  are  (or  single  program  is)  oosted. 

A  unique  cost  module  was  not  developed  for  this  methodology.  The  TAEG  16 
cost  model  developed  for  the  Navy  (Braby,  Henry,  et  al.,  1975)  was  used  to  cost 
the  media  and  methods.  As  discussed  earlier  this  model  costs  equipment  and 
methods  in  terms  of  37  variables  (input  to  this  computerized  model).  This  cost 
model  was,  at  the  time  of  Jorgensen  and  Hoffer' s  work, in  active  use  to  cost 
training  programs  at  the  Army  Air  Defense  School.  This  model  is  geared  to  in¬ 
stitutional  training  (although  adaptations  might  be  possible)  and  does  consider 
depreciation  and  LCC. 
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BDM's  GENERAL  MODEL/METHODOLOGY  FOR  CONDUCTING  COST  AND  TRAINING 

EFFECTIVENESS  ANALYSES  (1976) 


Vector  Research,  Inc.,  Cost  and  Training  Effectiveness  Analysis  (CTEA):  Handbook 
for  Action  Officers.  Leavenworth,' ftS:  EDM  Services  Company,  15  January  1976  .\ 

Vector  Research,  Inc.,  Developnent  of  a  Cost  and  Training  Effectivwiess  Analyti¬ 
cal  Models/Methodologles  for  Assessing  Training  Development  &  Products.  Leaven¬ 
worth,  KS:  EDM  Services  Company,  15  -January  1 976 . 

Vector  Research,  Inc.,  General  Model/Methodology  for  Conducting  Cost  and  Training 
Effectiveness  Analyses.  Leavenworth.  k£:  EDM  Services  Company,  id  bee ember  19 n. 


In  1976  Vector  Research,  Inc.,  working  as  a  subcontractor  to  BDM  Services 
Co.  prepared  a  CTEA  model/methodology  for  TRADOC.  BDM  saw  the  training  system 
as  a  number  of  subsystems,  each  to  be  assessed  and  costed;  and  an  evaluation 
made  of  the  aggregate  total  program  (Fig.  B-11). 

As  shown  in  Fig.  B-11,  it  was  based  on  a  list  of  critical  tasks  and  sug¬ 
gested  that  the  CTEA  study  team  consider  individual  and  collective  training  in 
both  unit  and  institution  and  by  several  methods.  However,  when  BDM  examined 
training  for  a  number  of  Army  systems  they  found  a  minimum  of  eight  unique  sys¬ 
tems.  These  they  identified  as: 

Large  Group  War  Games  Simulator 

Individualized  and/or  Small  Group  Lesson  Delivery  System 

Small  Group  Tactical  Maneuver  and  Deployment  Instructional  Game 

Program  Directed  Hands-On  Job  Performance  Aid 

Large  Weapon  System  Practice  Firing  Adapter 

Small  Group  Combat  Engagement  Simulator 

Trouble  Shooting  Training  Simulator 

Small  Weapon  System  Practice  Firing  Adapter 

They  then  proceeded  to  integrate  these  eight  types  into  their  general  model. 

The  general  model  is  shown  in  Fig.  B-12.  The  authors  of  the  methodology  also 
prepared  a  CTEA  handbook  for  action  officers  not  formally  trained  for  or  experi¬ 
enced  in  performing  CTEA.  In  spite  of  comprehensive  overview  shown  in  Fig.  B-11, 
the  guidebook  offers  no  assistance  to  the  action  officer  for  the  front-end  analy¬ 
sis.  He/ she  is  directed  to  prepare  a  simple  written  description  of  the  system 
being  discussed,  its  purpose  and  general  requirements.  Then  the  guidebook  directs 
the  officer  to  the  chpater  discussing  the  system  (of  the  eight)  most  like  the  one 
he/ she  has  identified. 

Although  the  model  (Fig.  B-12)  shows  a  concurrent  costing  and  effectiveness 
measurement  effort,  the  description  of  detailed  steps  for  each  type  system 
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Figure  B-ll.  Training  System  Assessment  Workshee 


Source  BDM.p.  Ill 


Figure  B-12.  CTEA  Flow  Model 


directs  the  analyst  to  address  cost  analysis  after  the  effectiveness  of  alter¬ 
native  systems  has  been  determined.  In  each  case  the  cost  analysis  method  is 
the  same.  The  analyst  selects  the  sample  system  most  similar  to  the  one(s)  to 
be  evaluated  and  then  proceeds.  Pour  forms  and  a  checklist  have  been  d*- signed 
to  ensure  that  all  necessary  steps  in  the  CTEA  have  been  completed. 

In  a  method  analogous  to  selection  of  a  measure  of  training  effectiveness 
(MOTE),  the  analyst  sets  a  standard  measurement  of  cost.  (This  always  follows 
the  determination  of  weighted  MOTEs  for  the  base  case  and  all  alternatives . ) 
Table  B-4  displays  examples  of  standard  measurement  of  cost  and  relevant  cost 
elements  for  each  of  the  eight  training  system  types.  When  the  analyst  has 
defined  the  standard  measure  of  cost,  he/she  enters  this  at  the  bottom  of  CTEA 
Form  #2  and  enters  identification  information  at  top  of  CTEA  Form  #3.  Next, 
the  appropriate  coat  elements  are  selected  and  their  LCC  cost  estimated  (also 
entered  on  Form  #3).  Suggested  sources  for  these  oost  data  are  the  Comptroller 
of  the  Army,  the  proponent  school,  or  CTEA  aotion  officer  estimation.  Costs 
are  sunned  for  each  training  system,  and  a  per  unit  oost  (based  on  Expected  LC 
number  of  units  of  system  used)  calculated.  These  per  unit  costs  are  then 
multiplied  by  the  effectiveness  ratio  and  compared  to  the  baseline  coat  and 
each  other  in  a  cost-benefit  analysis.  This  completes  the  CTEA. 
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Table  B-4.  Examples  of  Standards  of  Measurement  of  Cost  and  Relevant  Cost 
Elements  for  BDM  CTEA  Methodology's  Eight  Training  System  Types 
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Table  B-4  Examples  of  Standards  of  Measurement  of  Cost  and  Relevant  Cost 
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Table  B-4.  Examples  of  Standards  of  Measurement  of  Cost  and  Relevant  Cost 
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Elements  for  BDM  CTEA  Methodology's  Eight  Training  System  Types 


MODI A:  A  METHOD  OF  DESIGNING  INSTRUCTIONAL  ALTERNATIVES 
FOR  AIR  FORCE  TRAINING  (1977) 


(Carpenter-Hufftuan,  P.  MODIA;  Vol.l .  Overview  of  a  tool  for  planning  the  use 
of  Air  Force  training  resources,  R-ltOu.  Project  Air  Force  Office  (AF-RD<iiA)t 
Washington,  D'ff:  H3q  USA?,  July  1977) 

(Carpenter-Huffman,  P.  MODIA;  Vol.  2,  Options  for  oourse  design,  R-1701, 
Project  Air  Force  Office  (AF-RDQA),  Washington,  DC:  Hq  USAF,  July  1977) 


In  the  early  1970s  the  Rand  Corporation  undertook  an  extensive  effort  for 
the  Air  Force  to  develop  a  methodology  for  the  design  and  cost  analysis  of  an 
instructional  system.  MODIA  is  designed  as  a  tool  to  help  the  AF  manage  re¬ 
sources  for  formal  training  by  ’’systematically  and  explicitly  relating  quanti¬ 
tative  requirements  for  training  resources  to  the  details  of  course  design  and 
course  operation  during  the  planning  stage.”  (Carpenter-Huffman,  1977,  p.  2) 
MODIA  has  been  computerized  and  is  designed  specifically  for  use  in  the  five 
Air  Training  Command  (ATC)  technical  schools  that  account  for  90$  of  the  train¬ 
ing  load. 

The  computerized  cost  component  of  MODIA  is  called  MODCOM.  It  operates 
from  input  selected  by  the  user  based  on  course  operation  reports  output  from 
the  Resource  Utilization  Model  (RUM).  MODIA-designed  alternatives  will  vary  in 
both  cost  and  effectiveness.  They  may  then  be  compared  with  absolute  standards 
such  as  budget  constraints,  minimum  required  effectiveness,  and  with  each  other. 

Initial  input  to  MODIA  is  the  product  of  subject  matter  experts  in  planning 
and  developing  courses.  Their  path  in  definition  of  a  training  plan  is  not 
specified.  In  MODGOM,  some  of  the  requisite  cost  and  manning  factors  are  stored 
in  the  program  while  others  are  supplied  by  the  user  or  planner.  Total  course 
costs  for  up  to  five  years  are  included.  Options  on  the  included  costs  and  their 
method  of  computation  are  available.  The  specific  manpower  and  cost  categories 
for  which  estimates  are  developed  are: 

Manpower 

Students 
Instructors 
Curriculum  Personnel 
Hardware  Maintenance  Personnel 
Facilities  Maintenance  Personnel 
Trailing  Administrative  Personnel 
Base  Operating  Support  Personnel 
Medical  Personnel 

Cost  Categories 

Courseware  Procurement 
Hardware  Procurement 
Facility  Construction 
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Pay  and  Allowances 
Students 
Instructors 
Support  Personnel 

Permanent  Change  of  Station  (PCS) /Temporary  Duty  (TDY) 

Instructor  Training 
Miscellaneous  Operating  Expenses 

Alternatives  that  may  be  examined  are  changes  in  the  number  of  entrants; 
course  duration;  grade  structure  of  the  instructor  force;  the  levels  of  staff 
and  base  support;  pay  and  allowance  factors;  PCS/TDY  factors;  and  the  types  of 
courseware  and  hardware  used. 

The  author  states  that,  although  designed  for  use  with  MODIA,  MODCOM  may 
be  operated  independently.  It  produces  five  outputs; 

graduates  by  student  type 
student  and  staff  man-years 

courseware,  hardware,  and  facility  characteristics 

total  course  costs  by  functional  element 

total  course  costs  by  program  and  appropriation 

MODCOM  was  designed  to  include  ’relevant*  costs,  that  is,  only 
those  costs  which  are  both  incremental  (lie  in  the  future)  and 
variable  (with  respect  to  the  decision  to  offer  a  particular 
course) .  Costs  incurred  as  the  result  of  past  decisions  and 
costs  unaffected  by  the  existence  or  size  of  the  course  under 
analysis  are  not  considered  relevant  to  the  selection  of  an 
optimum  course  design. 

Since  it  is  intended  for  comparative  purposes,  all  costs  are  in  constant 
dollars.  Inherited  and  residual  value  of  resources  are  considered.  It  is  not 
suitable  for  short-range  budgetary  planning,  nonresident  training,  or  courses 
given  on  an  intermittent  basis. 

A  large  number  of  cost  data  elements  are  input  by  the  user  —  mostly  aver¬ 
age  costs  and  rates  or  factors.  Some  such  as  personnel  costs  are  calculated 
from  such  input  as  grade  and  pay  scale  matrices.  Sample  sources  are  shown  in 
Table  B-5. 

Output  samples  are  shewn  at  Annex  IV.  Course  costs  are  displayed  as  investment 
(courseware  and/or  hardware  procurement  and  facility  construction)  and  operating 
(pay  and  allowances,  rrilitary/civilian  personnel;  PCS;  TDY;  instructor  training; 
and  miscellaneous.  The  final  output  translates  costs  into  program  and  budget 
appropriation  cost  categories  as  defined  by  their  functional  cost  elements. 


TaJpl*  b-5.  Suggoatod  Coat  Data  Sourcoa ,  HODIA 


SOURCES: 

1.  Kaaslar  Technical  Training  Cantar,  Keaaler  AFB,  Mluippi 

2.  USAF  Cost  tod  Phoning  Factors,  AF  R  1 73*10,  Dapartmant 
of  tha  Air  Forca,  Headquarters  USAF,  Waahington,  O.C. 

3.  Rand-darived  aitimata(i). 

4.  Rand  estimate*  baaad  on  data  in  THa  Audio-  Vlauai  Equipment 
Directory.  National  Audio-Visual  Aatoeiation,  Inc., 

Fairfax,  Va. 

6.  Comptroller,  Headquartara  ATC,  Randolph  AFB,  Taxaa 

6.  EPIEGRAM  (newsletter  of  tha  Educational  Product* 
Information  Exchange  Inttituta,  N.Y.),  No.  13, 

April  1,  1973 

7.  DCS  Plan*.  Directorate  of  Manpower  and  Organiaation. 
Headquarters  ATC,  Randolph  AFB,  Taxaa 

8.  A  TC  Cost  Factors  Summary,  Dapartmant  of  tha  Air  Forca, 
Headquarters  ATC,  Randolph  AFB,  Taxaa 

9.  OMB  Circular  No.  A-94,  "Discount  ratat  to  be  used  in  evalu¬ 
ating  time-distributed  coats  and  benefit*, "  March  27,  1972, 
Office  of  Management  and  Budget,  Washington.  D.C. 


COORDINATION  OF  HUMAN  RESOURCE  TECHNOLOGIES  (CHRT)  (1978) 


Goclowski,  J.C. ,  King,  G.F.,  Ronoo,  P.G.,  &  Askren,  W.B.  Integration  and  Appli¬ 
cation  of  Hunan  Resource  Technologies  in  Weapon  System  Design:  Coordination  o? 
Kve  Hunan  Resource  Technologies.  AfrHpX-TTwS-S  (l)  brooks  Ai^6,  B(:  Q.S.  Air 
Force  Systems  Command,  l9tS.  T 

Goclowski,  J.C.,  King,  G.F.,  Ronco,  P.C.,  &  Askren,  W.B.  Integration  and  Appli¬ 
cation  of  Human  Resource  Technologies  in  Weapon  System  Design;  Processes  for 
the  Coordinated  Application  of  Five  Hunan  Resource  Technologies.  AFHrl-YR-76-6 
(ll)  Brooks  A?B,  TX:  0737” Air  Force  Systems  Cornnand ,  March  1 978 . 

Goclowski,  J.C.,  King,  G.F.,  Ronoo,  P.G.,  &  Askren,  W.B.  Integration  and  Appli¬ 
cation  of  Hunan  Resource  Technologies  in  Weapon  System  Design?  Consolidated  Data 
^ase  Functional  Specification.  AFHRL-tR-tfc-o  (111)  Brooks  A^B,  f)C:  U.S.  Air 
Force  Systems  Command,  May  i978. 


Goclowski  and  his  associates,  in  work  conducted  for  the  Air  Force  Human 
Resources  Laboratory  at- Wright-Pstterson  Air  Force  Case,  OH,  have  developed 
(1978)  a  methodology  for  coordinating  five  human  resource  technologies  (mainte¬ 
nance  manpower  modeling  (MW),  instructional  system  development  (ISD),  job  guide 
development  (JGD),  system  ownership  costing  (SGC),  and  human  resources  in  design 
trade-offs  (HRDT).  This  methodology  (CHRT)  quantified  the  reliability,  maintain¬ 
ability,  manpower,  training  and  JGD  requirements  for  a  weapon  system.  It  was 
designed  for  use  by  planners  early  in  the  system  acquisition  and  brings  the  in¬ 
fluence  of  the  aforementioned  quantified  factors  to  bear  on  design,  maintenance, 
operations  and  support  concepts  and  permits  estimation  of  ownership  costs. 

CHRT,  termed  a  predictive  and  product-oriented  methodology,  depends  on  four 
basic  activities. 

1.  Development  of  a  consolidated  data  base. 

2.  The  integrated  requirements  and  task  analysis. 

3.  ISD  and  JGD  product  development. 

4.  The  impact  analysis. 

Task  requirements  are  to  be  developed  very  early;  a  traditional,  integrated 
task  analysis,  during  full  scale  development.  CHRT  may  be  reiterated  for  alter¬ 
native  approaches.  IS  and  JG  products  follow  the  lost  task  analysis.  The  com¬ 
puterized  methodology  was  to  be  tested  using  data  from  the  advanced  medium  STOL 
transport  (AMST)  and  the  methodology  refined.  (A  report  on  this  work  has  not 
been  identified.) 

An  objective  of  this  development  was  the  production  of  weapon  system  opera¬ 
ting  and  support  costs.  To  this  end,  the  method  is  subordinate  to  the  ILS.  The 
ILS  plan,  properly  executed,  is  cited  as  the  primary  data  source  for  ten  sub¬ 
ordinate  categories,  including  (inter  alia)  personnel  and  training.  However,  the 
detailed  data  required  for  the  logistic  support  analysis  (LSA)  do  not  become 
available  until  the  full-scale  development  phase.  The  CHRT  process  is  planned 
to  fill  a  similar  need  for  the  conceptual  and  validation  phases  and  support  the 
detailed  design  in  later  phases. 


The  costing  portion  of  CHRT,  the  SOC,  was  not  fully  developed  at  the  tine 
of  this  report.  The  authors  state  that  more  comprehensive  models  are  needed  to 
identify  the  "real  ownership  cost  drivers."  As  developed,  the  SOC  technique 
builds  on  data  gathered  for  or  derived  from  operation  of  the  MMM,  ISD  and  JGD 
models.  A  flow  diagram  of  SOC  is  shown  in  Fig.  B— 13 .  When  the  trade-off  por¬ 
tion  of  the  methodology  is  exercised,  costs  may  be  constrained.  The  authors 
recognize  that  equation  and  models  for  obtaining  cost  estimates  are  not  stan¬ 
dardized  and  that  sources  of  data  do  not  always  adequately  reflect  the  system 
under  consideration. 

Task  is  basic  to  the  CHRT.  It  is  defined  as  "an  action  or  reaction  related 
to  equipment,  e.g.,  operate  aircraft,  remove  radio,  replace  transistor.  The 
conclusion  that  should  be  drawn  from  the  example  is  that  a  task  may  be  either 
general  or  very  specific  and  that  most  tasks  imply  subtasks."  (Goclowski  et  al., 
1978  (a)) 

Task  is  more  precisely  defined  in  Vol.  II  as:  "A  composite  of  related  activ¬ 
ities  (behaviors)  performed  by  an  individual  and  directed  toward  accomplishing  a 
specific  amount  of  work  within  a  specific  work  context.  These  activities  usually 
occur  in  temporal  proximity  with  the  same  displays  and  controls  and  have  a  common 
purpose.  Each  task  has  a  goal." 

General  data  requirements  for  the  SOC  are  shown  in  Table  b-6  along  with  other 
essential  data  for  exercising  the  CHRT.  Forty-one  items  (of  42  data  items  and 
27  sub- items)  are  shown  applicable  to  the  SOC.  Most  of  these  are  derived  from 
MMM,  ISD,  and  JGD  (26  items)  and  others  are  input  to  HRD  (9  items).  Those  unique 
to  SOC  are  unit  cost  goals,  D-T-G  goals,  R/W  consideration,  and  a  number  of  costs 
(manuals,  LRU  spares,  aircrew,  fuel,  depot  repairs,  faciities,  inventory  techni¬ 
cal  records  data,  on/off  equipment  maintenance,  and  maintenance  material.  As 
may  be  seen,  SOC  derives  costs  for  more  than  the  training  system.  SOC  is  de¬ 
signed  to  translate  human  resource  data  to  cost  data  on  a  system  or  subsystem 
basis . 

In  the  conceptual  and  validation  phase  the  task  list  is  obtained  from  gen¬ 
eral  maintenance  requirements,  a  knowledge  of  equipment  comparability,  mainte¬ 
nance  event  data,  maintenance  activity  data,  interviews  with  maintenance  person¬ 
nel,  comparable  tech  data  and  training  resources,  operations  and  mission  concepts 
and  plans,  operations  requirements,  comparable  training  courses  and  a  review  of 
existing  training  resources.  Data  are  separated  into  maintenance  and  operational 
categories  and  are  stored  in  task  condition  matrices. 

In  the  full  scale  development  phase,  a  general  task  analysis  is  supposed  to 
be  conducted  for  both  maintenance  and  operations  and  following  ISD  and  JGD,  a 
detailed  task  analysis. 

It  is  anticipated  that  through  experience  cost  estimating  formulae  for  the 
SOC  will  be  developed.  An  example  is  shown  in  Annex  VI(1).  The  cost  model  (a 
LCCM)  is  shown  in  Fig.  B— 14 .  The  authors  caveat  that  the  SOC  must  be  reviewei 
for  applicability  to  each  weapon  system  where  it  is  used  and  if  necessary  tail¬ 
ored  to  fit  a  particular  system  during  the  conceptual  phase.  It  requires  numeri¬ 
cal  data  input  for  each  element  (Annex  V(2)).  The  suggested  sources  are  standard 
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Table  B-6.  Data  Requirements  for  SOC  (Continued) 


HUMAN  RESOURCE  TECHNOLOGY  DATA 


DATA  (TIM 


HADTT 


26. 

Safety 

Hazards 

27. 

AvalfabUr  Ptnonntl  1 

a. 

Yean  of  Service 

b. 

Labor  Rate 

c. 

Scores 

d. 

Retention  Rate 

0. 

Predictions 

28 

Task  Freauency 

23. 

Task  Criticality 

30. 

Ta«k  Difficulty 

31. 

Degree  of  Proceduralization 

32. 

Content  of  Talk  Information 

33. 

Job  Guide  Concapt 

34. 

Job  Guide  Statui 

36. 

Manual  Content 

36. 

Training  Concapt 

37. 

Training  Statui 

38. 

Courla  Content 

|  39. 

Time  to  Train 

4°. 

Quantity  to  Train 

4*. 

Training  Resources 

42. 

Cost 

; 

a. 

SE  Investment 

b. 

Manual  Investment 

c. 

LRU  Spares  Investment 

d. 

Aircrew 

e. 

Fuel 

f. 

Depot  Repairs 

-  9- 

Facilities 

h. 

i. 

Inventory 

Technical  Record  Oata 

i- 

On/Off  Equipment  Maintenance 

k. 

Training 

1. 

Maintenance  Material 

ilivr’-'W  •i" " 1  -  . .  * 


igure  B-13.  System  Ownership  Cost  Flow  Diagram 


SYSTEM  OWNERSHIP  COST*  - 

SUPPORT  INVESTMENT  COSTS*  +  OPERATING  COSTS*  +  SUPPORT  COSTS* 


SUPPORT  INVESTMENT  COSTS*  •  CSE  +  CJG  +  CLS 

C$E  -  COST  OF  SUPPORT  EQUIPMENT 
CJG  -  COST  OF  JOB  GUIDES 
CLS  -  COST  OF  LRU  SPARES 


O&S  OPERATING  COSTS  -  Cpc  +  Cfc 

CPC  -  COST  OF  AIRCREW 
CFC  -  COST  OF  FUEL 


O&S  SUPPORT  COSTS  *  Cqr  +  +  CpA  +  C|m  +  ^TR  +  ^EM  +  Cpy 

CDp  *  COST  OF  DEPOT  REPAIRS 

CFA  -  COST  OF  FACILITIES 

Cim  -  COST  OF  INVENTORY  MANAGEMENT 

CTR  -  COST  OF  TECHNICAL  RECORD  DATA 

CEM  -  COST  OF  ON-OFF  EQUIPMENT  MAINTENANCE 

CpT  -  COST  OF  PERSONNEL  TRAINING 

*AII  costs  ftxprtsseri  in  annual  dollars,  i.a.,  dollars/yaar. 


Figure  B-14.  The  System  Ownership  Cost  Model  for  CHRT 
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government  offices/ documents  such  as  pay  and  allowance  tables,  cost  factor  man¬ 
uals  and  regulations.  Equations  for  each  cost  shown  in  Fig.  B-14  are  included  in 
Goclowski  et  al.T  1978  (c).  As  outlined  above,  the  task  list  creates  the  train¬ 
ing  program  with  its  requirements  for  job  guides,  personnel  training,  media,  etc. 
and  these  are  costed  along  with  all  other  support  costs.  Thus,  although  the 
life  cycle  training  costs  are  included  and  calculated,  they  do  not  appear  as 
unique  items  in  the  final  output. 


AN  AIR  DEFENSE  CTEA  METHODOLOGY  (1978) 


Hawley,  J.K.  and  Thomason,  S.C.  Development  of  an  Air  Defense  Co3t  and  Training 
Effectiveness  Analysis  (CTEA)  Methodology  (for  the  AN/TSQ-73) :  Vol  I  -  CTEA 
Within  the  Life  cycle  System  Management  Model .  Alexandria,  VA:  U.S.  Army  Re¬ 
search  Institute  for  the  Behavioral  and  Social  Sciences,  18  December  1978. 

_ ,  Vol.  II,  Demonstration  CTEA  on  the  AN/TSQ-73. 


Earlier  we  referred  to  a  methodology  developed  by  Hawley  and  Thomason  (1978). 
This  conceptual  model  was  designed  to  perform  CTEA  at  the  points  required  by  the 
LCSMM.  It  was  partially  demonstrated  on  an  existing  system  (Hawley  and  Thomason, 
1978a).  The  research  documentation  caveats  that  detailed  procedures  and  analysis 
techniques  must  be  developed  for  each  of  the  points  in  the  LCSMM  to  produce  a 
'full  blown  process'  model  from  the  conceptual  model. 

Hawley  and  Thomason  prepared  both  a  CTEA  methodology  and  list  of  events/ 
stages  of  the  LCSW.  Then  they  integrated  these  two  series  of  activities  since 
they  are  concurrent  processes.  In  their  opinion  this  integration  is  necessary 
since  hardware-related  and  training-related  activities  are  so  interactive  that 
'imbalances  in  development  can  easily  cause  CTEA  to  become  a  relatively  futile 
gesture'.  These  analysts  also  start  with  a  list  of  tasks  or  activities  generated 
from  analytical  study  of  the  materiel  system  hardware.  They  state  that,  for 
conceptual  systems,  an  activities  list  is  developed  as  part  of  the  training  input 
to  the  letter  of  agreement  (LOA).  After  the  development  of  the  hardware  proto¬ 
types,  the  task  list  is  obtained  from  the  Job/Task  Analysis;  the  job  conditions, 
from  the  Mission  Analysis;  the  standards,  empirically  or  judgmentally  derived. 

The  next  important  step  in  this  methodology  is  the  formulation  of  EEA  to 
assess  effectiveness  of  the  training  system.  For  each  EEA,  methods  of  training 
effectiveness  (MOTE)  and  measures  of  resource  requirements  (MTRR)  are  to  be 
specified.  The  MTRR,  the  resources  reouired  to  reach  the  training  objectives, 
constitute  the  cost  model  input. 

Examples  of  categories  of  MTRR  are  the  following: 

1 .  Operating  requirements  for  a  proposed  training  system  or  methodology: 

o  Instructors 
o  Students 
o  Facilities 

2.  Support  requirements: 

o  Personnel 
o  Materiel 
o  Supplies  consumed 
o  Parts  expended 

3.  The  reliability,  availability,  and  maintainability  (RAM)  of  the 
training  equipment.  Included  in  this  assessment  is  the  maintenance 
policy  for  complex  training  devices. 
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Suggested  sources  for  existing  data  are: 


1.  Past  studies  and  performance  data  (e.g.,  DT/OT,  SQT,  ARTEP,  etc.) 
on  similar  or  ancestral  systems. 

2.  Completed  tests  (e.g.,  DT/OT)  on  the  subject  system. 

3.  Task/ function  analysis  data. 

4.  Training  test  scores. 

5.  SQT  and  ARTEP  results. 

6.  Secondary  data  sources, 
o  Literature 

o  Institutional  data  bases 
o  Expert  opinion/judgment 
■\ 

The  analyst  is  advised  to  be  aware  that: 

1 .  Training  methodology  must  be  specified  at  a  level  of  detail  that 
can  be  reliably  costed. 

2.  Cost  estimates  should  include  initial  (set-up  or  sunk)  costs  as 
well  as  recurring  costs. 

3.  Some  COEA  may  require  training  LCCE  (especially  true  if  a  large 
initial  investment  is  to  be  made  in  equipment  subject  to  deprecia¬ 
tion  and/or  obsolescence) , 

Input  cost  data  are  categorized  as  falling  into  these  categories: 

1.  Facilities  —  land,  buildings,  etc. 

2.  Equipment  -  training  and  support 

3.  Instructional  material  -  development  and  acquisition 

4.  Personnel  -  instructors  and  training  support 

5 .  Students 

6 .  Supplies 

7.  Miscellaneous  (e.g.,  travel) 

Hawley  and  Thomason  also  recognize  the  importance  of  the  training  concept 
i'o  the  cost.  They  advise  the  cost  analyst  to  include  in  his  cost  data  requests, 
at  a  minimum,  the  following: 

1 .  The  training  concept  —  a  description  of  how  the  training  will 
be  carried  out,  instructional  methods  to  be  employed,  and  so 
forth . 

2.  The  make,  model,  quantity,  and  quality  of  materiel  required. 

3.  The  quantity,  grade,  and  skill  level  of  personnel  required. 
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4.  The  time  requirements  and  materiel  usage  data  —  the  length  of 
oourse,  facilities  required,  supplies  expended,  and  so  forth. 

They  state  that  training  resources  cost  quotes  should  be  expressed  in  con¬ 
stant  year-dollar's  and  that  DOD  policy  specifies  that  for  CTEA  the  base  cost 
year  is  the  fiscal  year  following  the  calendar  year  in  which  the  study  is 
scheduled  o  be  completed.  That  is,  the  base  cost  year  is  the  year  the  full- 
scale  or  limited  production  of  the  materiel  system  is  projected  to  begin  (e.g., 
between  ASARC  III  and  IOC).  (A  reference  for  ths  information  is  not  cited.) 

Other  costs  projections  recommended  are: 

1.  Sunk  oosts:  Cost  invested  in  tangible  assets  that  can  be 
recovered  only  by  use  of  the  assets  over  their  service  life. 

2.  Fixed  costs:  Costs  tnat  are  constant  in  total  amount  regardless 
of  production  volume,  e.g.,  depreciation  in  building  space 
occupied. 

3.  Variable  costs:  Costs  that  tend  to  vary  proportionately  to  the 
activity  in  the  function  involved,  e.g.,  direct  materials,  direct 
labor,  operating  supplies,  and  so  forth. 

4.  Semi variable  Costs:  Costs  having  a  fixed  and  a  variable  compo¬ 
nent,  such  as  the  cost  of  maintenance  and  repair  on  a  shut  down 
facility. 

They  also  say  that  provision  should  be  made  for  including  LCCE  to  project 
the  cost  of  the  training  program  over  the  life  of  the  materiel  system  (or  over 
some  reasonable  portion  of  it).  The  life  cycle  cost  methodology  should  also 
include  the  capability  for  discounting  training  cost  over  the  life  of  the  mate¬ 
riel  system. 

This  research  holds  that  the  accounting  and  discounting  requirements  are 
generally  available  in  existing  cost  models  (e.g.,  the  TECEP  costing  model  pre¬ 
sented  in  TAEG  Report  No.  1)  and  that  in  the  absence  of  special  cost  analysis 
requirements,  one  of  the  existing  procedures  be  used.  They  warn,  however,  that 
the  nature  of  the  available  training  cost  data  must  match  the  input  requirements 
of  the  costing  model.  Obversely,  if  the  EEA  require  features  not  available  in 
the  "canned"  model,  it  must  be  adapted  or  not  be  used.  This  methodology  also 
recommends  that  specific  comparisons  can  be  made  to  highlight  critical  cost 
considerations  and  cost-related  issues  that  require  further  exploration  in  the 
form  of  sensitivity  analyses  identified.  They  advise  sensitivity  analyses,  de¬ 
fined  as  investigations  of  the  effect  on  training  cost  and  resources  if  certain 
assumed  parameters  take  on  different  values.  As  examples  of  such  parameters 
they  cite: 

1.  Training  equipment  RAM  characteristics. 

2.  Instructor1  salaries  (e.g.,  converting  from  military  to  civil 
service  instructors  or  from  enlisted  personnel  to  officers) . 

3.  Student  selection  factors  (e.g.,  recruiting  a  larger  proportion 
of  category  III  or  IV  students  resulting  in  a  higher  attrition 
rate  or  more  remedial  time) . 
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The  training  coat  sensitivity  analyses  are  conducted  to  identify  particu¬ 
larly  sensitive  parameters  or  assumed  values  so  that  special  care  can  be  taken 
in  estimating  these  parameters  more  closely.  The  objective  is  to  select  train¬ 
ing  methodologies  that  are  cost-effective  for  most  of  the  range  of  likely  values 
for  these  assumed  parameters. 

In  a  second  volume  reporting  this  research  (Hawley  and  Thomason,  1978(b)), 
the  authors  attempted  to  exercise  the  methodology  using  data  from  a  developed 
system  (the  AN/TSQ-73).  A  number  of  problems  (believed  typical)  were  found  — 
LCSMM  not  strictly  adhered  to:  lack  of  coordination  between  hardware  system 
development  (SD)  and  ISD  (far  behind  HDS);  formal  task  list/analysis  non-existent 
TM  in  draft  and  full  of  errors.  The  HSD  had  passed  through  a  third  DT/OT  and 
it  was  too  late  for  CTEA  to  be  conducted  for  input  to  the  final  COEA.  Other 
attendant  problems  were  present. 

Because  of  the  ’primitive'  state  of  ISD,  the  researchers  chose  to  use 
Jorgensen  and  Hoffer's  (1978)  training  consonance  method  (an  estimate  of  com¬ 
patibility  of  task  and  training,  theoretically  correlated  with  training  effec¬ 
tiveness:  two  training  courses  —  the  Army's  and  the  contractor's  --  were 
proposed  and  were  the  candidates  for  exercising  the  CTEA  methodology. 

The  cost  portion  of  the  CTEA  was  not  exercised.  Programmed  instruction 
for  one  proposed  system  was  the  only  suggested  difference.  Although  found  to 
be  of  higher  acquisition  cost,  this  material  was  of  insignificant  cost  when 
amortized  over  the  life  of  the  system. 
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LIST  OF  ABBREVIATIONS  AND  ACRONYMS 


AD 

AIT 

AP 

APM 

AR 

ARTEP 

ASARC 

ASI 

ATM 

BDM/CARAF 

BOIP 

BOIPT 

BT 

CDB 

C/E 

CFP 

CHRT 

COEA 

CTEA 

DA 

DARCOM 

DCP 

DRIMS 

DSARC 

DT 

DT/OT 

DTD 

FYDP 

HRDT 

ILS 

IPR 

IPS 

IRTA 

ISD 

ISP 

ITC 

ITDT 

ITP 

I  TV 

JGD 

LCC 

LCCE 

LCSW 

LOA 

LOGCEN 

LR 

LSA 

LSAR 

MENS 

MILPERCEN 

m\ 

MOS 


Advanced  Development 
Advanced  Indiviaual  Training 
Acquisition  Plan 
Army  Program  Memorandum 
Army  Regulation 

Army  Training  and  Evaluation  Program 
Army  Systems  Acquisition  Review  Council 
Additional  Skill  Identifier 
Analogous  Task  Method 

The  BDM  Service  Company  Combined  Arms  Research  and 
Analysis  Facility 
Basis  of  Issue  Plan 
Basis  of  Issue  Plan  -  Tentative 
Basic  Training 
Consolidated  Data  Base 
Cost  Effectiveness 
Concept  Formulation  Package 
Coordinated  Human  Resources  Technology 
Cost  and  Operational  Effectiveness  Analysis 
Cost  and  Training  Effectiveness  Analysis 
Department  of  the  Army 

U.S.  Army  Materiel  Development  and  Readiness  Command 
Decision  Coordinating  Paper 
Diagnostic  Rifle  Marksmanship  Simulators 
Defense  System  Acquisition  Review  Council 
Development  Test 

Development  Test/Operational  Test 

Directorate  of  Training  Developments 

Five  Year  Defense  Plan 

Human  Resources  in  Design  Trade-Offs 

Integrated  Logistic  Support 

In-Process  Review 

Integrated  Personnel  Support 

Integrated  Requirements  and  Tasks  Analysis 

Instructional  Systems  Development 

Integrated  Support  Plan 

Institutional  Training  Cost 

Integrated  Technical  Documentation  and  Training 

Individual  Training  Plan 

Improved  TOW  Vehicle 

Job  Guide  Development 

Life  Cycle  Costs 

Life  Cycle  Cost  Elements 

Life  Cycle  System  Management  Model 

Letter  of  Agreement 

Logistics  Center 

Letter  Requirement 

Logistic  Support  Analysis 

Logistic  Support  Analysis  Record 

Mission  Element  Needs  Statement 

Military  Personnel  Center 

Maintenance  Manpower  Modeling 

Military  Occupational  Specialty 
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MODIA  Method  of  Designing  Instruction  Alternatives 

MOD  Modification 

MOTE  Measures  of  Training  EffectivenessNET 

MPA  Military  Personnel,  Army 

NET  New  Equipment  Training 

OAP  Outline  Acquisition  Plan 

ODP  Outline  .Development  Plan 

OICTP  Outline  Individual  Collective  Training  Plan 

OJT  On-the-job  Training 

OMA  Operations  and  Maintenance,  Army 

OSUT  One  Station  Unit  Training 

OT  Operational  Test 

Ph  Probability  of  Hit 

PIP  Product  Improvement  Program 

PM  Project  Manager 

POI  Program  of  Instruction 

PQQPRI  Provisional  Qualitative  and  Quantitative  Personnel 

Requirements  Information 
PROC  Procurement 

QQPRI  Qualitative  and  Quantitative  Personnel  Requirements 

Information 

RAM  Reliability,  Availability,  Maintainability 

RDTE  Research,  Development,  Test  and  Evaluation 

RFA  Rimfire  Adapter 

ROC  Required  Operational  Capability 

RTE  Relative  Training  Effectiveness 

RUM  Resource  Utilization  Model 

SAT  Systems  Approach  to  Training 

SOC  System  Ownership  Cost 

SQT  Skill  Qualification  Test 

TA2A  Task  and  Skill  Analysis 

TAEG  Training  Analysis  and  Evaluation  Group 

TCA  Training  Consonance  Analysis 

TECEP  Training  Evaluation  Cost  Evaluation  Program 

TDDA  Training  Developers  Decision  Aid 

TDIS  Training  Developments  Information  System 

TEEM  Training  Efficiency  Estimation  Model 

TM  Technical  Manual 

TCCI  Training  Option  Cost  Indicator 

TOE  Table  of  Organization  and  Equipment 

TQQPRI  Tentative  Qualitative  and  Quantitative  Personnel 

Requirements  Information 
TRADOC  Training  and  Doctrine  Command 

TRAINVICE  Training  Device  Effectiveness  Model 

TRAM  Training  Analysis  Model 

TIIAMCD  Training  Requirements  Analysis  Model 

TSD  Training  System  Design 

TSM  TRADOC  System  Manager 

TTC  Total  Training  Cost 

UI  User  Interface 

UTC  Unit  Training  Cost 

WSTEA  Weapon  System  Effectiveness  Analysis 
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